Simulate batch review

Overview

Splits a dataset that already has ground truth into an accepted part and a review part at random, copying the images and their annotations as they are. It stands in for a labeling and review pass in a demo or a test flow, where labeling every image by hand would take too long.

Review Percent of the images go to the review dataset and the rest to the accepted one. Wire the accepted dataset where a Review step’s accepted images would go, and the review dataset where its rejected ones would.

How it works

Takes every image and video of From Dataset, or a random sample of at most Max assets to process of them.

Shuffles them and sends Review Percent of them to the review dataset and the rest to the accepted dataset.

Adds the assets and their annotations to each dataset, skipping any the dataset already holds. The source dataset keeps everything.

FAQ

Where do the results go when I leave the destinations empty?

Each run creates a new dataset for each empty destination, named after the run, such as Accepted (simulated, task ID:...). Pick existing datasets to collect several runs in one place.

Why does the run fail saying a split would be empty?

Both parts must get at least one asset. With few assets, a small Review Percent rounds down to zero review assets, and 100% leaves nothing to accept. Raise or lower the percent, or use a larger dataset. An empty source dataset fails too.

Is the split the same on every run?

No. The sample and the split are random on every run, so running it again picks different assets.

Inputs

From Dataset

datasetRequired

Source dataset with existing ground truth annotations to split. Key: SRC_DATASET.

Review Percent

integerDefaults to 5

Percentage of assets randomly routed to the review dataset. The remaining assets go to the accepted dataset. Minimum: 1. Maximum: 100. Step: 1. Key: REVIEW_PERCENT.

Max assets to process

integer

Caps how many source assets are considered in this run. A random sample of at most this many assets is taken from the source dataset before the accepted/review split. Leave empty for no limit. Minimum: 1. Step: 1. Key: MAX_ASSETS.

Accepted dataset

dataset

Optional accepted dataset. If not provided, a new dataset is created. Key: ACCEPTED_DATASET.

Review dataset

dataset

Optional review dataset. If not provided, a new dataset is created. Key: REVIEW_DATASET.

Outputs

Accepted Dataset

dataset

ID of the dataset holding the assets simulated as accepted. Shown as an artifact. Key: ACCEPTED_DATASET.

Review Dataset

dataset

ID of the dataset holding the assets simulated as sent to review. Shown as an artifact. Key: REVIEW_DATASET.

JSON config

Machine-readable node interface for automation and advanced usage.

Show JSON

{
  "name": "Simulate batch review",
  "description": "Randomly splits a dataset that already has ground truth annotations into accepted and review datasets, copying assets and annotations as-is. Stands in for a full labeling and review pass on a demo where only a few images are labeled by hand.",
  "category": "Sample",
  "namespace": "ovalbee",
  "templateKey": "sample/simulate_batch_review",
  "version": "v1",
  "inputs": [
    {
      "key": "SRC_DATASET",
      "label": "From Dataset",
      "type": "dataset",
      "description": "Source dataset with existing ground truth annotations to split.",
      "required": true,
      "default": null,
      "visibleWhen": null
    },
    {
      "key": "REVIEW_PERCENT",
      "label": "Review Percent",
      "type": "number",
      "description": "Percentage of assets randomly routed to the review dataset. The remaining assets go to the accepted dataset.",
      "required": false,
      "default": 5,
      "visibleWhen": null,
      "options": {
        "type": "integer",
        "min": 1,
        "max": 100,
        "step": 1
      }
    },
    {
      "key": "MAX_ASSETS",
      "label": "Max assets to process",
      "type": "number",
      "description": "Caps how many source assets are considered in this run. A random sample of at most this many assets is taken from the source dataset before the accepted/review split. Leave empty for no limit.",
      "required": false,
      "default": null,
      "visibleWhen": null,
      "options": {
        "type": "integer",
        "min": 1,
        "step": 1
      }
    },
    {
      "key": "ACCEPTED_DATASET",
      "label": "Accepted dataset",
      "type": "dataset",
      "description": "Optional accepted dataset. If not provided, a new dataset is created.",
      "required": false,
      "default": null,
      "visibleWhen": null
    },
    {
      "key": "REVIEW_DATASET",
      "label": "Review dataset",
      "type": "dataset",
      "description": "Optional review dataset. If not provided, a new dataset is created.",
      "required": false,
      "default": null,
      "visibleWhen": null
    }
  ],
  "outputs": [
    {
      "key": "ACCEPTED_DATASET",
      "label": "Accepted Dataset",
      "type": "dataset",
      "description": "ID of the dataset holding the assets simulated as accepted.",
      "artifact": true,
      "badge": false,
      "badgeOpens": null,
      "hidden": false
    },
    {
      "key": "REVIEW_DATASET",
      "label": "Review Dataset",
      "type": "dataset",
      "description": "ID of the dataset holding the assets simulated as sent to review.",
      "artifact": true,
      "badge": false,
      "badgeOpens": null,
      "hidden": false
    }
  ]
}