> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.ovalbee.com/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.ovalbee.com/_mcp/server. # Similar pairs > Find all similar image pairs inside a dataset or across a reference dataset. Built with DINOv3. Dockerfile > **Note** > > First run can take longer while the worker pulls the configured Docker image. ## Overview Finds every pair of similar images inside one dataset, or between a dataset and a reference dataset, and grades each pair as an exact duplicate, near-identical, or strongly similar by three cosine-similarity thresholds. Built with DINOv3: DINOv3 is one of the embedding models it compares images with, used under the [DINOv3 License](https://ai.meta.com/resources/models-and-libraries/dinov3-license/). Search one dataset to find copies worth removing, or compare it with a reference, such as a test set against the training set, to find split leakage. The candidates land in a review dataset: in single-dataset mode every duplicate except one kept image per group, in reference mode the dataset images with an exact or near-identical match in the reference. Strongly similar pairs are reported but never become candidates. The badge counts the pairs and opens the review dataset. The source datasets stay unchanged. The report can be passed to [Dataset insights](/node-library/nodes/dataset-ops-dataset-insights) through its **Findings from other nodes** input. ```mermaid flowchart LR n1["Dataset"] --> n2["AI index embeddings"] --> n3["Similar pairs"] ``` ## Prerequisites | Requirement | What you need | | -------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Image embeddings** | Every image of the dataset, and of the reference dataset when set, indexed by [AI index embeddings](/node-library/nodes/models-clip-ai-index-embeddings) with the model chosen here (CLIP, PE-Core, SigLIP 2, or DINOv3). A single missing image stops the run. | ## How it works Checks that every image in scope has an embedding in the index of the selected model. Compares every image with every other one, or with every reference image, and keeps each pair at or above the strong similarity threshold, graded by level. Joins exact and near-identical pairs into duplicate groups in single-dataset mode, keeping the image with the smallest ID in each group, or collects the leaked dataset images in reference mode. Copies the candidates, with their annotations, into the review dataset and builds the report. ## FAQ #### What does the report row limit change? Only the all-pairs gallery. Counts, duplicate groups, findings, and the review dataset always use the complete pair set. #### Why does the run stop with incomplete image embeddings? At least one image of the dataset or the reference has no embedding for the selected model. Run AI index embeddings on each dataset with that model, then retry. #### Which threshold values are accepted? The three thresholds must satisfy strong similarity at most near-identical at most exact duplicate, all between 0 and 1. Any other combination stops the run. #### What happens to the review dataset on a rerun? The first run creates it, and every rerun adds a new version to it, so earlier results stay in its history. #### What if the reference dataset is the same as the dataset? It is treated as a single-dataset search. ## Inputs **`Dataset`** `dataset` — required Dataset searched for similar images. Its images must already be indexed by an AI Index Embeddings node. Key: `DATASET`. --- **`Reference dataset`** `dataset` Second dataset every image is compared against, for example a test set checked against the training set. Leave empty to compare the dataset with itself. Key: `REFERENCE_DATASET`. --- **`Embedding model`** `select` — required, default: clip Model both datasets were indexed with. Choosing one they were not indexed with stops the run. Key: `EMBEDDING_MODEL`. Options: * CLIP (`clip`) * PE-Core (`pecore`) * SigLIP 2 (`siglip2`) * DINOv3 (`dinov3`) --- **`Exact duplicate threshold`** `number` — required, default: 0.99 Similarity at or above which a pair is treated as an exact duplicate: a byte-identical or near-identical copy of the same frame. Minimum: `0`. Maximum: `1`. Step: `0.01`. Key: `EXACT_THRESHOLD`. --- **`Near-identical threshold`** `number` — required, default: 0.95 Similarity at or above which a pair is treated as near-identical: a resize, crop, or compression variant of the same image. Minimum: `0`. Maximum: `1`. Step: `0.01`. Key: `NEAR_THRESHOLD`. --- **`Strong similarity threshold`** `number` — required, default: 0.85 Similarity a pair must reach to be reported at all: the same scene or a semantically similar image, but not the same shot. Minimum: `0`. Maximum: `1`. Step: `0.01`. Key: `SIMILAR_THRESHOLD`. --- **`Report row limit`** `integer` — required, default: 500 Maximum number of pairs listed in the report. Counts, duplicate groups, findings, and the output dataset stay complete. Minimum: `1`. Step: `1`. Key: `REPORT_ROW_LIMIT`. --- ## Outputs **`Report`** `report` Pair table, duplicate groups or split leakage, and findings. Shown as an artifact. Key: `REPORT_ID`. --- **`Total pair count`** `string` Human-readable count of all unique pairs meeting the lowest threshold. Shown on the node as a badge. Selecting the badge opens `DUPLICATE_ASSETS`. Key: `TOTAL_PAIR_COUNT`. --- **`Duplicate assets`** `dataset` Dataset containing removal or leakage-review candidates from Dataset. Shown as an artifact. Key: `DUPLICATE_ASSETS`. --- ## Report The report leads with a verdict, shows where the pairs sit against the thresholds, and lists duplicate groups, leaked images, and pairs as image galleries. #### What does the report contain? The overview states how many dataset images have an exact or near-identical copy and, in single-dataset mode, how many images keeping one per group would remove. Metric cards carry the images with a duplicate, removable copies, and duplicate groups in single-dataset mode, or the dataset, reference, and leaked image counts in reference mode, plus the pair count broken down by level. A collapsed block records the mode, model, the three thresholds, and whether the pair gallery was truncated. A similarity histogram bins every qualifying pair and colours each bar by its level, so a threshold that cuts through one population is visible. Selecting a bar opens the pairs in it side by side. In single-dataset mode a gallery shows every duplicate group by its keeper, largest first. In reference mode it shows the leaked dataset images, the ones with the most exact or near-identical matches first. Selecting a card opens the whole group or every match. An all-pairs gallery closes the report, most similar first, limited by the report row limit. Selecting a pair card compares the two images side by side. The report has no tables: every list is a gallery of the images themselves. ## JSON config Machine-readable node interface for automation and advanced usage. #### Show JSON ```json { "name": "Similar pairs", "description": "Find all similar image pairs inside a dataset or across a reference dataset. Built with DINOv3.", "category": "Models", "namespace": "ovalbee", "templateKey": "models/clip/similar_pairs", "version": "v1", "inputs": [ { "key": "DATASET", "label": "Dataset", "type": "dataset", "description": "Dataset searched for similar images. Its images must already be indexed by an AI Index Embeddings node.", "required": true, "default": null, "visibleWhen": null, "options": { "creatable": false } }, { "key": "REFERENCE_DATASET", "label": "Reference dataset", "type": "dataset", "description": "Second dataset every image is compared against, for example a test set checked against the training set. Leave empty to compare the dataset with itself.", "required": false, "default": null, "visibleWhen": null, "options": { "creatable": false } }, { "key": "EMBEDDING_MODEL", "label": "Embedding model", "type": "select", "description": "Model both datasets were indexed with. Choosing one they were not indexed with stops the run.", "required": true, "default": "clip", "visibleWhen": null, "options": { "options": [ { "label": "CLIP", "value": "clip" }, { "label": "PE-Core", "value": "pecore" }, { "label": "SigLIP 2", "value": "siglip2" }, { "label": "DINOv3", "value": "dinov3" } ] } }, { "key": "EXACT_THRESHOLD", "label": "Exact duplicate threshold", "type": "number", "description": "Similarity at or above which a pair is treated as an exact duplicate: a byte-identical or near-identical copy of the same frame.", "required": true, "default": 0.99, "visibleWhen": null, "options": { "type": "float", "min": 0, "max": 1, "step": 0.01 } }, { "key": "NEAR_THRESHOLD", "label": "Near-identical threshold", "type": "number", "description": "Similarity at or above which a pair is treated as near-identical: a resize, crop, or compression variant of the same image.", "required": true, "default": 0.95, "visibleWhen": null, "options": { "type": "float", "min": 0, "max": 1, "step": 0.01 } }, { "key": "SIMILAR_THRESHOLD", "label": "Strong similarity threshold", "type": "number", "description": "Similarity a pair must reach to be reported at all: the same scene or a semantically similar image, but not the same shot.", "required": true, "default": 0.85, "visibleWhen": null, "options": { "type": "float", "min": 0, "max": 1, "step": 0.01 } }, { "key": "REPORT_ROW_LIMIT", "label": "Report row limit", "type": "number", "description": "Maximum number of pairs listed in the report. Counts, duplicate groups, findings, and the output dataset stay complete.", "required": true, "default": 500, "visibleWhen": null, "options": { "type": "integer", "min": 1, "step": 1, "slider": false } } ], "outputs": [ { "key": "REPORT_ID", "label": "Report", "type": "report", "kind": "dataset_intelligence_findings_v1", "description": "Pair table, duplicate groups or split leakage, and findings.", "artifact": true, "badge": false, "badgeOpens": null, "hidden": false }, { "key": "TOTAL_PAIR_COUNT", "label": "Total pair count", "type": "string", "description": "Human-readable count of all unique pairs meeting the lowest threshold.", "artifact": false, "badge": true, "badgeOpens": "DUPLICATE_ASSETS", "hidden": false }, { "key": "DUPLICATE_ASSETS", "label": "Duplicate assets", "type": "dataset", "description": "Dataset containing removal or leakage-review candidates from Dataset.", "artifact": true, "badge": false, "badgeOpens": null, "hidden": false } ], "runtime": { "type": "docker", "devices": [ "cpu" ], "dockerImage": "ovalbee/node-clip-index:0.0.32" }, "automation": { "isolated": true, "requires_configuration": true, "manual_run": true, "discard_history": "on_success" }, "widgets": { "widget": { "id": "asset-preview", "settings": { "datasetId": { "type": "variable", "value": "self.outputs.DUPLICATE_ASSETS" } } }, "secondaryWidget": { "id": "report-preview", "settings": { "reportAsset": { "type": "variable", "value": "self.outputs.REPORT_ID" }, "name": { "type": "input", "value": "Open report" } } } } } ```