Node LibraryLabel

Review

Overview

Hands annotated images or videos from a source dataset to reviewers in small batches, and moves each asset a reviewer judges to the dataset of the verdict button they clicked.

Each row in Outcome statuses pairs a verdict name with a destination dataset, for example Accepted with one dataset and Rejected with another. The annotation tool opens read-only, so reviewers judge the annotations instead of editing them. Clicking a verdict moves the asset and its annotation to that verdict’s dataset and hands the reviewer the next asset. Pick an existing dataset for each status, or type a new name in the dataset picker to create one. Assets leave the source dataset while a reviewer holds them. To hand out work that still has to be annotated, use Labeling instead.

Once every required input is set, the node runs itself every 15 minutes to top up queues and take back unfinished work.

Source dataset Reviewer queue Outcome dataset of the clicked verdict

How it works

1

Checks the source dataset and the Classes ontology, and adds the classes being reviewed to each outcome dataset’s ontology.

2

Gives each person in Reviewers a private queue, and sends the queue of anyone removed from the list back to the source dataset.

3

Returns assets that have sat in a queue longer than Return unclaimed work after (minutes) to the source dataset.

4

Tops each queue up to Batch size from the source dataset, in the order the assets arrived there.

5

Moves an asset and its annotation to the verdict’s dataset as soon as the reviewer clicks a verdict button in the annotation tool, then refills that reviewer’s queue right away.

FAQ

Every queue is pinned to an ontology, so each verdict is recorded against it. The run fails when Classes names no ontology: nothing picked while the source dataset has none, class names written without an ontology, or an ontology deleted since it was picked.

Every status needs a name and a destination dataset. The run also fails when two statuses share a name (ignoring case), when two statuses point at the same dataset, when there are no statuses at all, or when a destination dataset no longer exists.

The buttons come from the node’s last run. After you change Outcome statuses, run the node or wait for the next scheduled run, then reload the tool.

Each run adds whichever of the classes being reviewed (the ticked ones, or all of them) the outcome dataset’s ontology lacks, as a new version, and pins the dataset to it. An outcome dataset with no ontology gets a new ontology named after its status, holding those classes.

The clock starts when an asset enters a reviewer’s queue, not at their last action. Expired assets are returned on the node’s next run, so one can stay up to 15 minutes past the limit. The returned asset joins the end of the source dataset’s line.

A queue refills when its reviewer submits a verdict and on each node run, and only from what the source dataset holds. Assets added to an empty source dataset reach waiting reviewers at the next scheduled run, or right away if you run the node yourself.

No. The tool is read-only for this node. Send assets that need changes to an outcome dataset that feeds a Labeling node.

Every asset still in a reviewer’s queue goes back to the source dataset. The outcome datasets keep what was already submitted.

No. The run fails when the source dataset and the reviewers’ queues hold both images and videos.

Inputs

Source dataset
datasetRequired

Source dataset the annotated assets to review are drawn from. Anything a reviewer leaves unsubmitted comes back here. Key: SOURCE_DATASET.

Outcome statuses
labeling_outcomesRequired

Add the verdict buttons shown in the review tool and choose the destination dataset for each. Flow variables aren’t accepted. Key: OUTCOMES.

Reviewers
usersRequired

Workspace members who review. Each gets a private queue filled from the source dataset. Key: LABELERS.

Classes
ontologyRequired

Vocabulary this step reviews in. Defaults to the source dataset’s ontology. Pick another or create one here, and optionally tick only part of it - unticked entries are hidden from the reviewer while the queue stays pinned to the whole vocabulary, so the annotations stay readable in it, and entries added to the ontology later are shown. Tick nothing to review in all of it. Values come from SOURCE_DATASET. Key: CLASSES.

Batch size
integerDefaults to 10Required

Number of assets a reviewer holds at once. Their queue is topped up to this after each verdict and on every node run. Minimum: 1. Key: BATCH_SIZE.

Return unclaimed work after (minutes)
integerDefaults to 120Required

Assets left unsubmitted in a reviewer’s queue longer than this go back to the source dataset, so nobody sits on work they never finished. Minimum: 1. Key: CLAIM_TTL_MINUTES.

Outputs

User queues
object

Internal map of user IDs to their hidden review queue datasets. Hidden from the flow editor. Key: USERS_DATASETS.

Outcome datasets
object

Internal list of allowed outcome dataset IDs and button labels. Hidden from the flow editor. Key: OUTCOME_DATASETS.

Status
object

Compact per-reviewer queue status table object. Hidden from the flow editor. Key: STATUS.

Widgets

The node shows a table with one row per reviewer and the number of assets currently in their queue. Open annotation tool opens your own queue, so it works only for someone listed in Reviewers, and only after the node has run once since they were added. Reviewers can also open their queue from the Labeling page. The tool opens the image or video interface to match what the queue holds.

JSON config

Machine-readable node interface for automation and advanced usage.

{
  "name": "Review",
  "description": "Hand annotated images or videos from a source dataset to reviewers in small batches. The annotation tool opens read-only, and each outcome status becomes a verdict button that moves the asset and its annotation to that status's dataset.",
  "category": "Label",
  "namespace": "ovalbee",
  "templateKey": "label/review",
  "version": "v2",
  "inputs": [
    {
      "key": "SOURCE_DATASET",
      "label": "Source dataset",
      "type": "dataset",
      "description": "Source dataset the annotated assets to review are drawn from. Anything a reviewer leaves unsubmitted comes back here.",
      "required": true,
      "default": null,
      "visibleWhen": null,
      "options": {
        "creatable": false
      }
    },
    {
      "key": "OUTCOMES",
      "label": "Outcome statuses",
      "type": "labeling_outcomes",
      "description": "Add the verdict buttons shown in the review tool and choose the destination dataset for each.",
      "required": true,
      "default": null,
      "visibleWhen": null,
      "options": {
        "allowVariables": false
      }
    },
    {
      "key": "LABELERS",
      "label": "Reviewers",
      "type": "users",
      "description": "Workspace members who review. Each gets a private queue filled from the source dataset.",
      "required": true,
      "default": null,
      "visibleWhen": null
    },
    {
      "key": "CLASSES",
      "label": "Classes",
      "type": "ontology",
      "description": "Vocabulary this step reviews in. Defaults to the source dataset's ontology. Pick another or create one here, and optionally tick only part of it - unticked entries are hidden from the reviewer while the queue stays pinned to the whole vocabulary, so the annotations stay readable in it, and entries added to the ontology later are shown. Tick nothing to review in all of it.",
      "required": true,
      "default": null,
      "visibleWhen": null,
      "options": {
        "ref": "SOURCE_DATASET",
        "selectable": true,
        "creatable": true
      }
    },
    {
      "key": "BATCH_SIZE",
      "label": "Batch size",
      "type": "number",
      "description": "Number of assets a reviewer holds at once. Their queue is topped up to this after each verdict and on every node run.",
      "required": true,
      "default": 10,
      "visibleWhen": null,
      "options": {
        "type": "integer",
        "min": 1
      }
    },
    {
      "key": "CLAIM_TTL_MINUTES",
      "label": "Return unclaimed work after (minutes)",
      "type": "number",
      "description": "Assets left unsubmitted in a reviewer's queue longer than this go back to the source dataset, so nobody sits on work they never finished.",
      "required": true,
      "default": 120,
      "visibleWhen": null,
      "options": {
        "type": "integer",
        "min": 1
      }
    }
  ],
  "outputs": [
    {
      "key": "USERS_DATASETS",
      "label": "User queues",
      "type": "object",
      "description": "Internal map of user IDs to their hidden review queue datasets.",
      "artifact": false,
      "badge": false,
      "badgeOpens": null,
      "hidden": true
    },
    {
      "key": "OUTCOME_DATASETS",
      "label": "Outcome datasets",
      "type": "object",
      "description": "Internal list of allowed outcome dataset IDs and button labels.",
      "artifact": false,
      "badge": false,
      "badgeOpens": null,
      "hidden": true
    },
    {
      "key": "STATUS",
      "label": "Status",
      "type": "object",
      "description": "Compact per-reviewer queue status table object.",
      "artifact": false,
      "badge": false,
      "badgeOpens": null,
      "hidden": true
    }
  ],
  "automation": {
    "requires_configuration": true,
    "schedule": "*/15 * * * *",
    "discard_history": "on_success"
  },
  "widgets": {
    "widget": {
      "id": "table",
      "settings": {
        "data": {
          "type": "variable",
          "value": "self.outputs.STATUS"
        }
      }
    },
    "secondaryWidget": {
      "id": "open-labeling-tool",
      "settings": {
        "name": {
          "type": "input",
          "value": "Open annotation tool"
        }
      }
    }
  }
}