Label diagnostics

Dockerfile

First run can take longer while the worker pulls the configured Docker image.

Overview

Finds annotated objects whose label may be wrong: objects that look unlike the rest of their own class, and classes whose objects look more like another class. It compares the embedding of every object with the other objects of its class.

The report lists both as review candidates, not confirmed mistakes. The images holding the suspect objects, with their annotations, go into a review dataset, and the badge counts the suspect objects and opens that dataset.

The report can be passed to Dataset insights through its Findings from other nodes input.

Prerequisites

Requirement	What you need
Object annotations	Images that already carry object annotations, with at least two objects per class you want checked.
Object embeddings	This dataset indexed by an AI index embeddings node pointed at it, with object indexing on and the model chosen here. DINOv3 also needs visual embedding on. Objects indexed only through another dataset do not count.

How it works

Reads the dataset’s annotated objects and checks that every one of them has an embedding in the index of the selected model.

Skips objects smaller than Minimum object size on either side, and classes left with fewer than two objects.

Measures how far each object sits from the rest of its class, and flags the most distant Outlier percentile of every class as suspects.

Checks which class each object sits nearest to, to find objects closer to another class than to their own.

Copies the images holding suspect objects, with their annotations, into the review dataset and builds the report. The input dataset is not changed.

FAQ

Why does every class have at least one suspect?

The percentile is taken per class and rounded up, so a class of 10 objects at 5% still gives one suspect. Read the distance, not only the list: a suspect sitting inside the main mass of the histogram is an ordinary object.

Why does the run stop with incomplete object embeddings?

The index does not hold every object the dataset’s annotations contain, usually because annotations changed since the last indexing run or the index node has object indexing off. Run AI index embeddings on this dataset with the same model, then retry.

What other errors stop the run?

A dataset with no annotated objects, a workspace with no index for the selected model, and a dataset where no class keeps two objects above the minimum size all stop the run with an error.

What happens to the review dataset on a rerun?

The first run creates it, and every rerun adds a new version to it, so earlier results stay in its history. Each suspect image arrives with its whole annotation, not only the suspect object.

Inputs

Annotated dataset

datasetRequired

Dataset whose object annotations are checked. Its objects must already be indexed by an AI Index Embeddings node. Key: DATASET.

Embedding model

selectDefaults to clipRequired

Model the dataset was indexed with. Choosing one it was not indexed with stops the run. Key: EMBEDDING_MODEL.

Options:

CLIP (clip)
PE-Core (pecore)
SigLIP 2 (siglip2)
DINOv3 (dinov3)

Outlier percentile

numberDefaults to 5Required

Percentage of each class’s objects reported as suspects, taking the ones least like the rest of their own class. Minimum: 0.01. Maximum: 100. Step: 0.01. Key: OUTLIER_PERCENTILE.

Minimum object size

integerDefaults to 32Required

Smallest object side in pixels that still takes part. Narrower or shorter objects are skipped, along with any class left with fewer than two objects. Minimum: 1. Step: 1. Key: MIN_OBJECT_SIZE.

Outputs

Report

report

Object outliers, directed class confusion, and source-image previews. Shown as an artifact. Key: REPORT_ID.

Suspect count

string

Human-readable suspect-object count. Selecting the badge opens their unique source images. Shown on the node as a badge. Selecting the badge opens SUSPECT_ASSETS. Key: SUSPECT_COUNT.

Suspect assets

dataset

Short-lived dataset with unique source images and suspect annotation assets. Shown as an artifact. Key: SUSPECT_ASSETS.

Report

The report leads with a verdict, shows which classes are confused and with what, plots every object’s distance to its own class, and opens source evidence with annotation overlays and bounding-box crops.

What does the report contain?

The overview states how many analyzed objects sit closer to another class than to their own and names the strongest confusion direction. Four metric cards carry the analyzed objects, included classes, label outliers, and possible confusions with their shares, and a collapsed block records the model, the settings, and the skipped-object count. A per-class chart splits every class into objects nearest their own class and objects nearest another, worst class first, with a per-class summary table below it. Class confusion is drawn as a heatmap of assigned class against nearest class, in shares of the assigned class, next to the directed table of off-diagonal candidates. The heatmap is left out when nothing is confused, and replaced by a notice above 200 classes. Label outliers are shown on a histogram of every object’s distance to the rest of its class, with the flagged objects in their own colour, above the ranked outlier table. Every chart segment, heatmap cell, and table row opens the objects behind it, cropped to their bounding boxes with the source annotation.

JSON config

Machine-readable node interface for automation and advanced usage.

Show JSON

{
  "name": "Label diagnostics",
  "description": "Find likely object-label outliers and directed class confusion in an annotated dataset.",
  "category": "Models",
  "namespace": "ovalbee",
  "templateKey": "models/clip/label_diagnostics",
  "version": "v1",
  "inputs": [
    {
      "key": "DATASET",
      "label": "Annotated dataset",
      "type": "dataset",
      "description": "Dataset whose object annotations are checked. Its objects must already be indexed by an AI Index Embeddings node.",
      "required": true,
      "default": null,
      "visibleWhen": null,
      "options": {
        "creatable": false
      }
    },
    {
      "key": "EMBEDDING_MODEL",
      "label": "Embedding model",
      "type": "select",
      "description": "Model the dataset was indexed with. Choosing one it was not indexed with stops the run.",
      "required": true,
      "default": "clip",
      "visibleWhen": null,
      "options": {
        "options": [
          {
            "label": "CLIP",
            "value": "clip"
          },
          {
            "label": "PE-Core",
            "value": "pecore"
          },
          {
            "label": "SigLIP 2",
            "value": "siglip2"
          },
          {
            "label": "DINOv3",
            "value": "dinov3"
          }
        ]
      }
    },
    {
      "key": "OUTLIER_PERCENTILE",
      "label": "Outlier percentile",
      "type": "number",
      "description": "Percentage of each class's objects reported as suspects, taking the ones least like the rest of their own class.",
      "required": true,
      "default": 5,
      "visibleWhen": null,
      "options": {
        "type": "float",
        "min": 0.01,
        "max": 100,
        "step": 0.01
      }
    },
    {
      "key": "MIN_OBJECT_SIZE",
      "label": "Minimum object size",
      "type": "number",
      "description": "Smallest object side in pixels that still takes part. Narrower or shorter objects are skipped, along with any class left with fewer than two objects.",
      "required": true,
      "default": 32,
      "visibleWhen": null,
      "options": {
        "type": "integer",
        "min": 1,
        "step": 1,
        "slider": false
      }
    }
  ],
  "outputs": [
    {
      "key": "REPORT_ID",
      "label": "Report",
      "type": "report",
      "kind": "dataset_intelligence_findings_v1",
      "description": "Object outliers, directed class confusion, and source-image previews.",
      "artifact": true,
      "badge": false,
      "badgeOpens": null,
      "hidden": false
    },
    {
      "key": "SUSPECT_COUNT",
      "label": "Suspect count",
      "type": "string",
      "description": "Human-readable suspect-object count. Selecting the badge opens their unique source images.",
      "artifact": false,
      "badge": true,
      "badgeOpens": "SUSPECT_ASSETS",
      "hidden": false
    },
    {
      "key": "SUSPECT_ASSETS",
      "label": "Suspect assets",
      "type": "dataset",
      "description": "Short-lived dataset with unique source images and suspect annotation assets.",
      "artifact": true,
      "badge": false,
      "badgeOpens": null,
      "hidden": false
    }
  ],
  "runtime": {
    "type": "docker",
    "requiresGpu": false,
    "dockerImage": "cr.internal.supervisely.com/ovalbee-internal/nodes/clip-index:0.0.24"
  },
  "automation": {
    "isolated": true,
    "requires_configuration": true,
    "manual_run": true,
    "discard_history": "on_success"
  },
  "widgets": {
    "widget": {
      "id": "report-preview",
      "settings": {
        "reportAsset": {
          "type": "variable",
          "value": "self.outputs.REPORT_ID"
        },
        "name": {
          "type": "input",
          "value": "Open diagnostics"
        }
      }
    }
  }
}