Skip to navigation
Node libraryLabel

Review

Overview

Manages your review work, like a team lead. You give it a dataset of labeled images or videos (assets) and a team of reviewers, and it runs the review for you:

  • creates a small personal queue for each reviewer, so nobody picks work by hand and two people never get the same asset;
  • opens the annotation tool read-only, so reviewers judge the annotations without changing them;
  • refills a queue as soon as its reviewer gives a verdict;
  • sends each checked asset to the right dataset, based on the verdict button the reviewer pressed;
  • takes back work that someone holds for too long, or that a removed reviewer left behind.

You decide which verdicts reviewers see. In Decision buttons, each row is a verdict name and the dataset it sends assets to. A new node starts with two: Accept and Reject. While an asset is in someone’s queue, it isn’t in the source dataset, so the source only holds work that nobody has taken yet.

Once every required input is set, the node runs by itself every 15 minutes to take back unfinished work. New assets in the source dataset don’t wait for that run: they go straight to queues with free space.

To hand out work that still needs labeling, use Labeling.

Source dataset Reviewer queue Outcome dataset of the clicked verdict

How it works

1

Creates a personal queue for each person in Reviewers and fills it up to Batch size from the source dataset, in the order the assets arrived there, or highest anomaly score first when Queue order asks for it.

2

Moves an asset and its annotation to the verdict’s dataset as soon as the reviewer presses a verdict button in the annotation tool. The reviewer’s queue refills right away.

3

Returns assets that stayed in a queue longer than Return unclaimed work after (minutes) to the source dataset.

4

Sends the queue of anyone you remove from Reviewers back to the source dataset.

5

Adds the classes being reviewed to each outcome dataset’s ontology.

FAQ

A verdict without a dataset gets a new one on the node’s first run. It’s named Accepted for Accept, Rejected for Reject, and after the verdict name for any other. The node then saves that dataset in the verdict’s row. You can also pick an existing dataset, or type a new name in the dataset picker to create one.

Each queue is tied to an ontology (the set of classes and tags), so every verdict is saved against it. The run fails when Classes has no ontology. This happens when nothing is picked and the source dataset has no ontology, when class names are typed without an ontology, or when the picked ontology was deleted.

Every button needs a name. The run also fails when:

  • two buttons have the same name (letter case doesn’t matter);
  • two buttons use the same dataset;
  • there are no buttons at all;
  • a destination dataset no longer exists.

The buttons come from the node’s last run. After you change Decision buttons, run the node or wait for the next scheduled run. Then reload the tool.

Each run looks for classes being reviewed that the outcome dataset’s ontology doesn’t have yet. These are the ticked classes, or all of them when nothing is ticked. The run adds them as a new ontology version and switches the dataset to it. An outcome dataset without an ontology gets the one picked in Classes. This way all outcome datasets share the same classes.

The timer starts when an asset enters a reviewer’s queue, not at their last action. The 15-minute run returns expired assets, so an asset can stay up to 15 minutes past the limit. The returned asset is placed after the assets already waiting in the source dataset.

A queue refills when its reviewer gives a verdict and when new assets arrive in the source dataset. It can only take what the source dataset has. The 15-minute run is a backup, not the normal way work arrives.

No. The tool is read-only for this node. To get annotations fixed, send those assets to an outcome dataset that feeds a Labeling node.

Every asset still in a reviewer’s queue goes back to the source dataset. The outcome datasets keep everything already judged.

No. The run fails when the source dataset and the reviewers’ queues hold both images and videos.

Inputs

Source dataset
datasetRequired

Dataset with the labeled assets to review. Assets a reviewer doesn’t judge come back here. Key: SOURCE_DATASET.

Decision buttons
labeling_outcomesDefaults to [{'id': 'accept', 'label': 'Accept', 'dataset': None}, {'id': 'reject', 'label': 'Reject', 'dataset': None}]Required

Buttons shown in the annotation tool, each with a destination dataset. Clicking a button moves the current item there, with its annotations. Flow variables aren’t accepted. Key: OUTCOMES.

Reviewers
usersRequired

Workspace members who review. Each person gets a personal queue from the source dataset. Key: LABELERS.

Classes
ontologyRequired

Classes and tags to review in. Defaults to the source dataset’s ontology. You can pick another one or create one here. Tick some entries to show only those to reviewers, or tick nothing to show all of them. The queue still uses the whole ontology, so the annotations stay valid in it, and entries added to the ontology later are shown. Values come from SOURCE_DATASET. Key: CLASSES.

Queue order
selectDefaults to arrivalRequired

Which assets go into a reviewer’s queue first, and the order the tool shows them in. Highest anomaly score first reads the score Embedding anomalies saves to each image, so the hardest cases come first. Images without a score follow, oldest first. Flow variables aren’t accepted. Key: QUEUE_ORDER.

Options:

  • Oldest first (arrival)
  • Highest anomaly score first (anomaly_score)
Batch size
integerDefaults to 10Required

How many assets a reviewer holds at once. The queue refills to this number after each verdict and on every node run. Minimum: 1. Key: BATCH_SIZE.

Return unclaimed work after (minutes)
integerDefaults to 120Required

Assets that stay in a reviewer’s queue longer than this without a verdict go back to the source dataset. This keeps unfinished work from getting stuck. Minimum: 1. Key: CLAIM_TTL_MINUTES.

Outputs

User queues
object

Internal map of user IDs to their hidden review queue datasets. Hidden from the flow editor. Key: USERS_DATASETS.

Outcome datasets
object

Internal list of allowed outcome dataset IDs and button labels. Hidden from the flow editor. Key: OUTCOME_DATASETS.

Status
object

Compact per-reviewer queue status table object. Hidden from the flow editor. Key: STATUS.

Widgets

The node card shows a live chart of the work: assets in progress in reviewers’ queues and how many assets they accepted or rejected. Below the chart is a list of reviewers with the number of items each one reviewed. Click a reviewer to see only their numbers. The chart updates a few seconds after every Accept or Reject, without a node run.

Open annotation tool opens your own queue. It works only if you’re in Reviewers, and only after the node has run once since you were added. Reviewers can also open their queue from the Labeling page. The tool opens the image or video interface to match what the queue holds.

JSON config

Machine-readable node interface for automation and advanced usage.

{
  "name": "Review",
  "description": "People check labeled images or videos here. They can't change the labels. A button, such as Accept or Reject, sends the item to its dataset.",
  "category": "Label",
  "namespace": "ovalbee",
  "templateKey": "label/review",
  "version": "v2",
  "inputs": [
    {
      "key": "SOURCE_DATASET",
      "label": "Source dataset",
      "type": "dataset",
      "description": "Dataset with the labeled assets to review. Assets a reviewer doesn't judge come back here.",
      "required": true,
      "default": null,
      "visibleWhen": null,
      "options": {
        "creatable": false
      }
    },
    {
      "key": "OUTCOMES",
      "label": "Decision buttons",
      "type": "labeling_outcomes",
      "description": "Buttons shown in the annotation tool, each with a destination dataset. Clicking a button moves the current item there, with its annotations.",
      "required": true,
      "default": [
        {
          "id": "accept",
          "label": "Accept",
          "dataset": null
        },
        {
          "id": "reject",
          "label": "Reject",
          "dataset": null
        }
      ],
      "visibleWhen": null,
      "options": {
        "allowVariables": false
      }
    },
    {
      "key": "LABELERS",
      "label": "Reviewers",
      "type": "users",
      "description": "Workspace members who review. Each person gets a personal queue from the source dataset.",
      "required": true,
      "default": null,
      "visibleWhen": null
    },
    {
      "key": "CLASSES",
      "label": "Classes",
      "type": "ontology",
      "description": "Classes and tags to review in. Defaults to the source dataset's ontology. You can pick another one or create one here. Tick some entries to show only those to reviewers, or tick nothing to show all of them. The queue still uses the whole ontology, so the annotations stay valid in it, and entries added to the ontology later are shown.",
      "required": true,
      "default": null,
      "visibleWhen": null,
      "options": {
        "ref": "SOURCE_DATASET",
        "selectable": true,
        "creatable": true
      }
    },
    {
      "key": "QUEUE_ORDER",
      "label": "Queue order",
      "type": "select",
      "description": "Which assets go into a reviewer's queue first, and the order the tool shows them in. Highest anomaly score first reads the score Embedding anomalies saves to each image, so the hardest cases come first. Images without a score follow, oldest first.",
      "required": true,
      "default": "arrival",
      "visibleWhen": null,
      "options": {
        "allowVariables": false,
        "options": [
          {
            "label": "Oldest first",
            "value": "arrival"
          },
          {
            "label": "Highest anomaly score first",
            "value": "anomaly_score"
          }
        ]
      }
    },
    {
      "key": "BATCH_SIZE",
      "label": "Batch size",
      "type": "number",
      "description": "How many assets a reviewer holds at once. The queue refills to this number after each verdict and on every node run.",
      "required": true,
      "default": 10,
      "visibleWhen": null,
      "options": {
        "type": "integer",
        "min": 1
      }
    },
    {
      "key": "CLAIM_TTL_MINUTES",
      "label": "Return unclaimed work after (minutes)",
      "type": "number",
      "description": "Assets that stay in a reviewer's queue longer than this without a verdict go back to the source dataset. This keeps unfinished work from getting stuck.",
      "required": true,
      "default": 120,
      "visibleWhen": null,
      "options": {
        "type": "integer",
        "min": 1
      }
    }
  ],
  "outputs": [
    {
      "key": "USERS_DATASETS",
      "label": "User queues",
      "type": "object",
      "description": "Internal map of user IDs to their hidden review queue datasets.",
      "artifact": false,
      "badge": false,
      "badgeOpens": null,
      "hidden": true
    },
    {
      "key": "OUTCOME_DATASETS",
      "label": "Outcome datasets",
      "type": "object",
      "description": "Internal list of allowed outcome dataset IDs and button labels.",
      "artifact": false,
      "badge": false,
      "badgeOpens": null,
      "hidden": true
    },
    {
      "key": "STATUS",
      "label": "Status",
      "type": "object",
      "description": "Compact per-reviewer queue status table object.",
      "artifact": false,
      "badge": false,
      "badgeOpens": null,
      "hidden": true
    }
  ],
  "automation": {
    "requires_configuration": true,
    "schedule": "*/15 * * * *",
    "discard_history": "on_success"
  },
  "widgets": {
    "widget": {
      "id": "labeling-stats",
      "settings": {}
    },
    "secondaryWidget": {
      "id": "open-labeling-tool",
      "settings": {
        "name": {
          "type": "input",
          "value": "Open annotation tool"
        }
      }
    }
  }
}