Node LibraryModels

Similar pairs

Dockerfile
First run can take longer while the worker pulls the configured Docker image.

Overview

Finds every pair of similar images inside one dataset, or between a dataset and a reference dataset, and grades each pair as an exact duplicate, near-identical, or strongly similar by three cosine-similarity thresholds.

Search one dataset to find copies worth removing, or compare it with a reference, such as a test set against the training set, to find split leakage. The candidates land in a review dataset: in single-dataset mode every duplicate except one kept image per group, in reference mode the dataset images with an exact or near-identical match in the reference. Strongly similar pairs are reported but never become candidates. The badge counts the pairs and opens the review dataset. The source datasets stay unchanged.

The report can be passed to Dataset insights through its Findings from other nodes input.

Dataset AI index embeddings Similar pairs

Prerequisites

RequirementWhat you need
Image embeddingsEvery image of the dataset, and of the reference dataset when set, indexed by AI index embeddings with the model chosen here (CLIP, PE-Core, SigLIP 2, or DINOv3). A single missing image stops the run.

How it works

1

Checks that every image in scope has an embedding in the index of the selected model.

2

Compares every image with every other one, or with every reference image, and keeps each pair at or above the strong similarity threshold, graded by level.

3

Joins exact and near-identical pairs into duplicate groups in single-dataset mode, keeping the image with the smallest ID in each group, or collects the leaked dataset images in reference mode.

4

Copies the candidates, with their annotations, into the review dataset and builds the report.

FAQ

Only the all-pairs gallery. Counts, duplicate groups, findings, and the review dataset always use the complete pair set.

At least one image of the dataset or the reference has no embedding for the selected model. Run AI index embeddings on each dataset with that model, then retry.

The three thresholds must satisfy strong similarity at most near-identical at most exact duplicate, all between 0 and 1. Any other combination stops the run.

The first run creates it, and every rerun adds a new version to it, so earlier results stay in its history.

It is treated as a single-dataset search.

Inputs

Dataset
datasetRequired

Dataset searched for similar images. Its images must already be indexed by an AI Index Embeddings node. Key: DATASET.

Reference dataset
dataset

Second dataset every image is compared against, for example a test set checked against the training set. Leave empty to compare the dataset with itself. Key: REFERENCE_DATASET.

Embedding model
selectDefaults to clipRequired

Model both datasets were indexed with. Choosing one they were not indexed with stops the run. Key: EMBEDDING_MODEL.

Options:

  • CLIP (clip)
  • PE-Core (pecore)
  • SigLIP 2 (siglip2)
  • DINOv3 (dinov3)
Exact duplicate threshold
numberDefaults to 0.99Required

Similarity at or above which a pair is treated as an exact duplicate: a byte-identical or near-identical copy of the same frame. Minimum: 0. Maximum: 1. Step: 0.01. Key: EXACT_THRESHOLD.

Near-identical threshold
numberDefaults to 0.95Required

Similarity at or above which a pair is treated as near-identical: a resize, crop, or compression variant of the same image. Minimum: 0. Maximum: 1. Step: 0.01. Key: NEAR_THRESHOLD.

Strong similarity threshold
numberDefaults to 0.85Required

Similarity a pair must reach to be reported at all: the same scene or a semantically similar image, but not the same shot. Minimum: 0. Maximum: 1. Step: 0.01. Key: SIMILAR_THRESHOLD.

Report row limit
integerDefaults to 500Required

Maximum number of pairs listed in the report. Counts, duplicate groups, findings, and the output dataset stay complete. Minimum: 1. Step: 1. Key: REPORT_ROW_LIMIT.

Outputs

Report
report

Pair table, duplicate groups or split leakage, and findings. Shown as an artifact. Key: REPORT_ID.

Total pair count
string

Human-readable count of all unique pairs meeting the lowest threshold. Shown on the node as a badge. Selecting the badge opens DUPLICATE_ASSETS. Key: TOTAL_PAIR_COUNT.

Duplicate assets
dataset

Short-lived dataset containing removal or leakage-review candidates from Dataset. Shown as an artifact. Key: DUPLICATE_ASSETS.

Report

The report leads with a verdict, shows where the pairs sit against the thresholds, and lists duplicate groups, leaked images, and pairs as image galleries.

The overview states how many dataset images have an exact or near-identical copy and, in single-dataset mode, how many images keeping one per group would remove. Metric cards carry the images with a duplicate, removable copies, and duplicate groups in single-dataset mode, or the dataset, reference, and leaked image counts in reference mode, plus the pair count broken down by level. A collapsed block records the mode, model, the three thresholds, and whether the pair gallery was truncated. A similarity histogram bins every qualifying pair and colours each bar by its level, so a threshold that cuts through one population is visible. Selecting a bar opens the pairs in it side by side. In single-dataset mode a gallery shows every duplicate group by its keeper, largest first. In reference mode it shows the leaked dataset images, the ones with the most exact or near-identical matches first. Selecting a card opens the whole group or every match. An all-pairs gallery closes the report, most similar first, limited by the report row limit. Selecting a pair card compares the two images side by side. The report has no tables: every list is a gallery of the images themselves.

JSON config

Machine-readable node interface for automation and advanced usage.

{
  "name": "Similar pairs",
  "description": "Find all similar image pairs inside a dataset or across a reference dataset.",
  "category": "Models",
  "namespace": "ovalbee",
  "templateKey": "models/clip/similar_pairs",
  "version": "v1",
  "inputs": [
    {
      "key": "DATASET",
      "label": "Dataset",
      "type": "dataset",
      "description": "Dataset searched for similar images. Its images must already be indexed by an AI Index Embeddings node.",
      "required": true,
      "default": null,
      "visibleWhen": null,
      "options": {
        "creatable": false
      }
    },
    {
      "key": "REFERENCE_DATASET",
      "label": "Reference dataset",
      "type": "dataset",
      "description": "Second dataset every image is compared against, for example a test set checked against the training set. Leave empty to compare the dataset with itself.",
      "required": false,
      "default": null,
      "visibleWhen": null,
      "options": {
        "creatable": false
      }
    },
    {
      "key": "EMBEDDING_MODEL",
      "label": "Embedding model",
      "type": "select",
      "description": "Model both datasets were indexed with. Choosing one they were not indexed with stops the run.",
      "required": true,
      "default": "clip",
      "visibleWhen": null,
      "options": {
        "options": [
          {
            "label": "CLIP",
            "value": "clip"
          },
          {
            "label": "PE-Core",
            "value": "pecore"
          },
          {
            "label": "SigLIP 2",
            "value": "siglip2"
          },
          {
            "label": "DINOv3",
            "value": "dinov3"
          }
        ]
      }
    },
    {
      "key": "EXACT_THRESHOLD",
      "label": "Exact duplicate threshold",
      "type": "number",
      "description": "Similarity at or above which a pair is treated as an exact duplicate: a byte-identical or near-identical copy of the same frame.",
      "required": true,
      "default": 0.99,
      "visibleWhen": null,
      "options": {
        "type": "float",
        "min": 0,
        "max": 1,
        "step": 0.01
      }
    },
    {
      "key": "NEAR_THRESHOLD",
      "label": "Near-identical threshold",
      "type": "number",
      "description": "Similarity at or above which a pair is treated as near-identical: a resize, crop, or compression variant of the same image.",
      "required": true,
      "default": 0.95,
      "visibleWhen": null,
      "options": {
        "type": "float",
        "min": 0,
        "max": 1,
        "step": 0.01
      }
    },
    {
      "key": "SIMILAR_THRESHOLD",
      "label": "Strong similarity threshold",
      "type": "number",
      "description": "Similarity a pair must reach to be reported at all: the same scene or a semantically similar image, but not the same shot.",
      "required": true,
      "default": 0.85,
      "visibleWhen": null,
      "options": {
        "type": "float",
        "min": 0,
        "max": 1,
        "step": 0.01
      }
    },
    {
      "key": "REPORT_ROW_LIMIT",
      "label": "Report row limit",
      "type": "number",
      "description": "Maximum number of pairs listed in the report. Counts, duplicate groups, findings, and the output dataset stay complete.",
      "required": true,
      "default": 500,
      "visibleWhen": null,
      "options": {
        "type": "integer",
        "min": 1,
        "step": 1,
        "slider": false
      }
    }
  ],
  "outputs": [
    {
      "key": "REPORT_ID",
      "label": "Report",
      "type": "report",
      "kind": "dataset_intelligence_findings_v1",
      "description": "Pair table, duplicate groups or split leakage, and findings.",
      "artifact": true,
      "badge": false,
      "badgeOpens": null,
      "hidden": false
    },
    {
      "key": "TOTAL_PAIR_COUNT",
      "label": "Total pair count",
      "type": "string",
      "description": "Human-readable count of all unique pairs meeting the lowest threshold.",
      "artifact": false,
      "badge": true,
      "badgeOpens": "DUPLICATE_ASSETS",
      "hidden": false
    },
    {
      "key": "DUPLICATE_ASSETS",
      "label": "Duplicate assets",
      "type": "dataset",
      "description": "Short-lived dataset containing removal or leakage-review candidates from Dataset.",
      "artifact": true,
      "badge": false,
      "badgeOpens": null,
      "hidden": false
    }
  ],
  "runtime": {
    "type": "docker",
    "requiresGpu": false,
    "dockerImage": "cr.internal.supervisely.com/ovalbee-internal/nodes/clip-index:0.0.24"
  },
  "automation": {
    "isolated": true,
    "requires_configuration": true,
    "manual_run": true,
    "discard_history": "on_success"
  },
  "widgets": {
    "widget": {
      "id": "report-preview",
      "settings": {
        "reportAsset": {
          "type": "variable",
          "value": "self.outputs.REPORT_ID"
        },
        "name": {
          "type": "input",
          "value": "Open similar pairs"
        }
      }
    }
  }
}