Sample images similar to failures

Dockerfile

First run can take longer while the worker pulls the configured Docker image.

Overview

Closes the loop between evaluating a model and improving its training data. It reads which images the model scored worst on, then searches a pool of candidate images for ones that look like them — so the next round of labelling is spent on the material the model actually struggles with rather than on more of what it already handles.

It embeds nothing itself. The CLIP Index node has already turned every indexed image into a vector, and the worst images from the report are among them, so this node only queries.

Each rerun writes a new version of the node’s own hidden LAST_RUN dataset holding that run’s sample, and the badge shows how many images it selected and opens it. Set a destination dataset to accumulate samples across runs as well.

Prerequisites

Requirement	What you need
An evaluation report	An object detection report from the Evaluate node. Reports produced before this node existed do not carry the per-image file it reads, so re-run the evaluation if you see it complain about that.
An indexed pool	A CLIP Index node pointed at your candidate pool. It runs on a schedule, so give it a few minutes on a freshly wired dataset.

How it works

Ranks the evaluated images by the metric you choose and takes the worst few as seeds. Images with no ground-truth objects are skipped — a zero score on an empty frame says nothing about the model.

Asks the CLIP index which pool images sit nearest each seed, ignoring anything below the similarity threshold and anything in the Exclude images in dataset. Point that at your labeled set and the pool never has to be edited to keep picked images out of the next sample.

Folds the per-seed results together, keeping a candidate that is near several seeds once at its best score, and drops the seeds themselves along with anything the report already evaluated.

Inputs

Evaluation report

reportRequired

Object detection evaluation report the failures are read from. Generic reports are not supported. Key: REPORT_ID.

Candidate pool

datasetRequired

Dataset the similar images are drawn from. Its images must already be in the CLIP index, so run a CLIP Index node on it first. Key: POOL_DATASET.

Exclude images in

dataset

Images in this dataset are never sampled, so the pool itself never has to change. In an active learning loop, point it at the labeled training set and each sample holds only images nobody has picked yet. Key: EXCLUDE_DATASET.

To dataset

dataset

Destination for the sample. Leave empty to keep it only in the node’s own Last run dataset. Key: DST_DATASET.

Worst by

selectDefaults to f1Required

Which per-image metric decides the failures the sample is seeded from. Key: WORST_BY.

Options:

F1 (f1)
Recall (recall)
Precision (precision)

Sample size

integerDefaults to 100Required

Maximum number of images to select. Fewer are returned when the pool holds nothing else similar enough. Minimum: 1. Maximum: 10000. Step: 1. Key: SAMPLE_SIZE.

Failures used as seeds

integerDefaults to 25Required

How many of the worst-scoring evaluated images the search starts from. Minimum: 1. Maximum: 500. Step: 1. Key: SEED_COUNT.

Similarity threshold

numberDefaults to 0.25Required

Ignore candidates less similar than this to every seed. Raise it when the sample drifts off-topic. Minimum: 0. Maximum: 1. Step: 0.01. Key: SIMILARITY_THRESHOLD.

Exclude evaluated images

booleanDefaults to trueRequired

Leave out images the report already evaluated, so the sample is only new material. Key: EXCLUDE_EVALUATED.

Outputs

Dataset

dataset

Key: DATASET.

Last run

dataset

Hidden dataset holding this run’s sample. A rerun writes a new version of it. Shown as an artifact. Key: LAST_RUN.

Selected images

string

Number of image assets attached to the sample. Shown on the node as a badge. Selecting the badge opens LAST_RUN. Key: SELECTED_IMAGES.

JSON config

Machine-readable node interface for automation and advanced usage.

Show JSON

{
  "name": "Sample images similar to failures",
  "description": "Take the images an evaluation report scored worst and sample visually similar ones from a candidate pool.",
  "category": "Sample",
  "namespace": "ovalbee",
  "templateKey": "sample/sample_images_similar_to_failures",
  "version": "v1",
  "inputs": [
    {
      "key": "REPORT_ID",
      "label": "Evaluation report",
      "type": "report",
      "kind": "object_detection_evaluation_report_v1",
      "description": "Object detection evaluation report the failures are read from. Generic reports are not supported.",
      "required": true,
      "default": null,
      "visibleWhen": null
    },
    {
      "key": "POOL_DATASET",
      "label": "Candidate pool",
      "type": "dataset",
      "description": "Dataset the similar images are drawn from. Its images must already be in the CLIP index, so run a CLIP Index node on it first.",
      "required": true,
      "default": null,
      "visibleWhen": null
    },
    {
      "key": "EXCLUDE_DATASET",
      "label": "Exclude images in",
      "type": "dataset",
      "description": "Images in this dataset are never sampled, so the pool itself never has to change. In an active learning loop, point it at the labeled training set and each sample holds only images nobody has picked yet.",
      "required": false,
      "default": null,
      "visibleWhen": null
    },
    {
      "key": "DST_DATASET",
      "label": "To dataset",
      "type": "dataset",
      "description": "Destination for the sample. Leave empty to keep it only in the node's own Last run dataset.",
      "required": false,
      "default": null,
      "visibleWhen": null
    },
    {
      "key": "WORST_BY",
      "label": "Worst by",
      "type": "select",
      "description": "Which per-image metric decides the failures the sample is seeded from.",
      "required": true,
      "default": "f1",
      "visibleWhen": null,
      "options": {
        "control": "tabs",
        "options": [
          {
            "label": "F1",
            "value": "f1"
          },
          {
            "label": "Recall",
            "value": "recall"
          },
          {
            "label": "Precision",
            "value": "precision"
          }
        ]
      }
    },
    {
      "key": "SAMPLE_SIZE",
      "label": "Sample size",
      "type": "number",
      "description": "Maximum number of images to select. Fewer are returned when the pool holds nothing else similar enough.",
      "required": true,
      "default": 100,
      "visibleWhen": null,
      "options": {
        "type": "integer",
        "min": 1,
        "max": 10000,
        "step": 1,
        "slider": false
      }
    },
    {
      "key": "SEED_COUNT",
      "label": "Failures used as seeds",
      "type": "number",
      "description": "How many of the worst-scoring evaluated images the search starts from.",
      "required": true,
      "default": 25,
      "visibleWhen": null,
      "options": {
        "type": "integer",
        "min": 1,
        "max": 500,
        "step": 1,
        "slider": false
      }
    },
    {
      "key": "SIMILARITY_THRESHOLD",
      "label": "Similarity threshold",
      "type": "number",
      "description": "Ignore candidates less similar than this to every seed. Raise it when the sample drifts off-topic.",
      "required": true,
      "default": 0.25,
      "visibleWhen": null,
      "options": {
        "type": "float",
        "min": 0,
        "max": 1,
        "step": 0.01
      }
    },
    {
      "key": "EXCLUDE_EVALUATED",
      "label": "Exclude evaluated images",
      "type": "boolean",
      "description": "Leave out images the report already evaluated, so the sample is only new material.",
      "required": true,
      "default": true,
      "visibleWhen": null
    }
  ],
  "outputs": [
    {
      "key": "DATASET",
      "label": "Dataset",
      "type": "dataset",
      "description": "",
      "artifact": false,
      "badge": false,
      "badgeOpens": null,
      "hidden": false
    },
    {
      "key": "LAST_RUN",
      "label": "Last run",
      "type": "dataset",
      "description": "Hidden dataset holding this run's sample. A rerun writes a new version of it.",
      "artifact": true,
      "badge": false,
      "badgeOpens": null,
      "hidden": false
    },
    {
      "key": "SELECTED_IMAGES",
      "label": "Selected images",
      "type": "string",
      "description": "Number of image assets attached to the sample.",
      "artifact": false,
      "badge": true,
      "badgeOpens": "LAST_RUN",
      "hidden": false
    }
  ],
  "runtime": {
    "type": "docker",
    "devices": [
      "cpu"
    ],
    "dockerImage": "ovalbee/node-clip-index:0.0.32"
  },
  "automation": {
    "isolated": true,
    "requires_configuration": true,
    "manual_run": true,
    "discard_history": "on_success"
  }
}