Annotation quality

Overview

Checks the annotations of an image dataset and builds a quality report: how many images carry labels, how the classes balance, and which issues need a look, such as duplicate images, labels outside the frame, split leakage, and misspelled or rare classes.

Use it after importing, preparing, or sampling a dataset to see whether annotations, classes, tags, and image coverage look healthy before training or evaluation. The badge sums it up, and the report can be passed to Dataset insights through its Findings from other nodes input.

How it works

Scans every image in the dataset and its annotation.

Reuses the statistics already measured for annotations that have not changed, and measures the rest.

Compares the content fingerprints storage already holds for each image, to find the ones stored more than once. No image is downloaded for this.

Computes coverage, class, object-size, and tag statistics across the dataset.

Writes the class list and summary statistics onto the dataset and builds the report, opened by the preview widget.

FAQ

How are duplicate images found?

By content, not by name. Storage keeps a hash of every object it holds, so the check compares those and never downloads an image — importers routinely rename files, which is exactly why two copies of one picture can sit in a dataset unnoticed. The count is the copies that could be removed, so three copies of one picture count as two. The gallery shows every copy, and thumbnails sharing a caption are the same picture. Only byte-identical files are reported, so a resized or re-compressed copy is not a duplicate here and will not be listed. Where the hash cannot answer the question — an image uploaded in parts hashes differently depending on how it was split — those images are listed separately as unchecked rather than quietly reported as unique.

What counts as an annotated image?

An annotation is the record attached to an image, and it can hold objects (labeled shapes) and tags. An image counts as annotated only when it has at least one object: a bare or empty annotation record is not coverage.

Do image tags and object tags count as annotations?

A tag is part of an annotation, not a separate annotation. Objects, image-level tags, and object-level tags are analyzed as three separate dimensions. An image whose annotation carries only tags adds no objects, so it counts as an image without objects.

What does the badge report?

The share of images with objects, the class count, and the issue count when there are issues.

Why is a second run faster?

Statistics are stored per annotation, in a store Dataset insights reads too, so a rerun only downloads and measures annotations it has not seen. Over an unchanged dataset it measures nothing.

Does the node change my dataset?

Images and annotations are left as they are. The node writes the class list and summary statistics, such as image, object, and class counts and coverage, into the dataset’s metadata.

Which data can it analyze?

Image assets and their annotations only. Video assets are not analyzed.

Inputs

Dataset

datasetRequired

Dataset whose image annotations are checked. Measurements are kept, so a rerun over an unchanged dataset recomputes nothing. Key: DATASET.

Outputs

Report

report

Annotation quality report for the dataset. Shown as an artifact. Key: REPORT_ID.

Quality Summary

string

Compact dataset quality summary shown as a badge. Shown on the node as a badge. Key: QUALITY_SUMMARY.

Report

The report is attached to the node as an artifact and opens in the preview widget. Sections that have nothing to show are dropped rather than rendered empty, so a dataset without objects or tags produces a shorter report, and a dataset with no labels at all keeps little more than the Overview with a “No labels yet” notice.

Overview is the summary you read first: four metric cards for images (with the annotated share), objects per image, classes (with the first few names), and issues, graded by severity. An image counts as annotated only when it carries at least one object.

Issues lists what the statistics noticed, one row per issue rather than one per image, with a count and a gallery on click. Severity describes the dataset, never the run: Problem marks what cannot be right, while Check marks what is worth a look and may still be intentional.

Issue	Severity	What it means
Exact copy of another image	Problem	The dataset stores the same picture more than once, byte for byte, under different names. Counted as the copies that could be removed, so three copies of one picture count as two.
Could not be compared for duplicates	Check	Storage holds no usable hash for these images, typically because they were uploaded in parts, so a duplicate among them would go unreported.
Images in two splits	Problem	The same picture is in, say, train and test. Two images count as the same when the file name, dimensions, object count and classes all match.
Class labeled only outside train	Check	The class has no objects in the train split. If training respects the split tags, the model never learns this class. The node cannot know whether it does, so this is a check, not a problem. Raised only when the dataset has a train split.
Labels reaching outside the image frame	Problem	The bounding box starts before the frame or ends past it, by more than one pixel.
Labels with zero width or height	Problem	The box has no extent. Point, line and graph geometries are exempt, being flat by nature.
Duplicate labels of the same class	Problem	Two labels of one class overlap at a bounding-box IoU above 0.9 — the same object annotated twice.
Names spelled two ways	Problem	Two class names differ only in spacing or separators, splitting one class in two. Case-only pairs are left alone: a dataset may label `a` and `A` on purpose.
Possible misspelling	Check	A rare class name is one edit away from a far more frequent one and is not an English word itself, as `resestor` is next to `resistor`. Names that are real words are never reported: `bus` beside `bush` cannot be judged without context.
Rare class	Check	The class covers more than ten times fewer images than the median class of this dataset. Rarity is measured against the dataset, so a balanced vocabulary of many small classes stays quiet.
Images with no labels	Check	Counted in one row, never one per image, and left out when no image in the dataset has labels. Unlabeled images carrying the `split: test` asset tag are excluded — public datasets routinely ship their test split unlabeled — and the intro says how many were set aside.

Label size is deliberately not an issue: small and elongated labels are normal in aerial imagery, defect inspection and fibre segmentation. The section appears only when there is something to show.

Images starts with an image coverage pie and, when some image carries a tag, a bar chart of images by asset tag. That chart counts an image once per tag it has, so its bars can add up to more than the image count, and images with no tags of their own form an Untagged bar. Bookkeeping tags written by import and sampling nodes are left out — an import path or a sampler node id says how an image arrived rather than what it holds, and being unique per image it would fill the chart with one bar per image. Tags that describe the data, such as split and source_video, are shown. A chart of image resolutions and a table of every image with its annotation count, object count, class count, dimensions, classes, and labeled-area percentage follow. Selecting a row previews that image with its annotations.

Object sections

These appear only when the dataset contains objects.

Section	What it shows
Objects	Every parsed object with geometry type, bounding-box position and size, and area, next to a chart of objects by geometry type. Selecting a row previews the object cropped from its image.
Objects per image	An overall histogram of objects per image, plus a heatmap of the same counts broken down by class. Outliers show up as sparse high-count cells.
Class balance	Charts of objects by class and of images and objects by class for spotting imbalance at a glance, followed by sortable image coverage, object count, and average object area per class.
Class sizes	Object area, width, and height statistics per class, a min/avg/max area range chart, and an object-area histogram whose leftmost bin collects degenerate labels. Area is relative to image area, while width and height are the bounding box in pixels.
Class co-occurrence	How often two classes appear on the same image. The diagonal is how many images contain that class at all.
Spatial distribution	Heatmaps of bounding-box centers on a fixed 16 x 16 grid, for all classes together and for the four largest classes. Row 1, column 1 is the top-left of the frame. An even spread means no positional bias, while a strong hot spot means a location bias.

Charts are clickable: a bar, slice, or heatmap cell opens a gallery of the images behind it. Very large galleries are capped, and co-occurrence matrices are skipped with a notice when the dataset has more than 200 classes or tags to draw.

Tag sections

Image-level tag sections appear only when the dataset has image tags, and object-level ones only when it has object tags.

Section	What it shows
Image tag co-occurrence	How often image tags are used together on the same image.
Object tag co-occurrence	How often object tags are used together on the same object.
Class to tag co-occurrence	Which object tags go with which annotation class.
Categorical tags distribution	Tag values across the dataset, one section for image-level tags and one for object-level. Useful for spotting imbalance in how tags were assigned.

JSON config

Machine-readable node interface for automation and advanced usage.

Show JSON

{
  "name": "Annotation quality",
  "description": "Check the annotations of an image dataset and report coverage, class balance, and issues such as duplicate images, out-of-frame labels, and split leakage.",
  "category": "Datasets",
  "namespace": null,
  "templateKey": "dataset_ops/annotation_quality",
  "version": "v1",
  "inputs": [
    {
      "key": "DATASET",
      "label": "Dataset",
      "type": "dataset",
      "description": "Dataset whose image annotations are checked. Measurements are kept, so a rerun over an unchanged dataset recomputes nothing.",
      "required": true,
      "default": null,
      "visibleWhen": null,
      "options": {
        "creatable": false
      }
    }
  ],
  "outputs": [
    {
      "key": "REPORT_ID",
      "label": "Report",
      "type": "report",
      "kind": "dataset_intelligence_findings_v1",
      "description": "Annotation quality report for the dataset.",
      "artifact": true,
      "badge": false,
      "badgeOpens": null,
      "hidden": false
    },
    {
      "key": "QUALITY_SUMMARY",
      "label": "Quality Summary",
      "type": "string",
      "description": "Compact dataset quality summary shown as a badge.",
      "artifact": false,
      "badge": true,
      "badgeOpens": null,
      "hidden": false
    }
  ],
  "widgets": {
    "widget": {
      "id": "report-preview",
      "settings": {
        "reportAsset": {
          "type": "variable",
          "value": "self.outputs.REPORT_ID"
        },
        "name": {
          "type": "input",
          "value": "Open quality report"
        }
      }
    }
  }
}