> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.ovalbee.com/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.ovalbee.com/_mcp/server.

# Review

> People check labeled images or videos here. They can't change the labels. A button, such as Accept or Reject, sends the item to its dataset.

## Overview

Manages your review work, like a team lead. You give it a dataset of labeled images or videos (assets) and a team of reviewers, and it runs the review for you:

* creates a small personal queue for each reviewer, so nobody picks work by hand and two people never get the same asset;
* opens the annotation tool read-only, so reviewers judge the annotations without changing them;
* refills a queue as soon as its reviewer gives a verdict;
* sends each checked asset to the right dataset, based on the verdict button the reviewer pressed;
* takes back work that someone holds for too long, or that a removed reviewer left behind.

You decide which verdicts reviewers see. In **Decision buttons**, each row is a verdict name and the dataset it sends assets to. A new node starts with two: **Accept** and **Reject**. While an asset is in someone's queue, it isn't in the source dataset, so the source only holds work that nobody has taken yet.

Once every required input is set, the node runs by itself every 15 minutes to take back unfinished work. New assets in the source dataset don't wait for that run: they go straight to queues with free space.

To hand out work that still needs labeling, use [Labeling](/node-library/nodes/label-labeling).

```mermaid
flowchart LR
  source["Source dataset"] --> queue["Reviewer queue"] --> outcome["Outcome dataset of the clicked verdict"]
```

## How it works

Creates a personal queue for each person in **Reviewers** and fills it up to **Batch size** from the source dataset, in the order the assets arrived there, or highest anomaly score first when **Queue order** asks for it.

Moves an asset and its annotation to the verdict's dataset as soon as the reviewer presses a verdict button in the annotation tool. The reviewer's queue refills right away.

Returns assets that stayed in a queue longer than **Return unclaimed work after (minutes)** to the source dataset.

Sends the queue of anyone you remove from **Reviewers** back to the source dataset.

Adds the classes being reviewed to each outcome dataset's ontology.

## FAQ

#### Which datasets does a new node create?

A verdict without a dataset gets a new one on the node's first run. It's named **Accepted** for Accept, **Rejected** for Reject, and after the verdict name for any other. The node then saves that dataset in the verdict's row. You can also pick an existing dataset, or type a new name in the dataset picker to create one.

#### Why did the run fail with a Classes error?

Each queue is tied to an ontology (the set of classes and tags), so every verdict is saved against it. The run fails when **Classes** has no ontology. This happens when nothing is picked and the source dataset has no ontology, when class names are typed without an ontology, or when the picked ontology was deleted.

#### Why did the run fail on Decision buttons?

Every button needs a name. The run also fails when:

* two buttons have the same name (letter case doesn't matter);
* two buttons use the same dataset;
* there are no buttons at all;
* a destination dataset no longer exists.

#### Why doesn't a verdict I added show up in the annotation tool?

The buttons come from the node's last run. After you change **Decision buttons**, run the node or wait for the next scheduled run. Then reload the tool.

#### What happens to the ontology of an outcome dataset?

Each run looks for classes being reviewed that the outcome dataset's ontology doesn't have yet. These are the ticked classes, or all of them when nothing is ticked. The run adds them as a new ontology version and switches the dataset to it. An outcome dataset without an ontology gets the one picked in **Classes**. This way all outcome datasets share the same classes.

#### When does unfinished work go back to the source dataset?

The timer starts when an asset enters a reviewer's queue, not at their last action. The 15-minute run returns expired assets, so an asset can stay up to 15 minutes past the limit. The returned asset is placed after the assets already waiting in the source dataset.

#### Why is a reviewer's queue not full?

A queue refills when its reviewer gives a verdict and when new assets arrive in the source dataset. It can only take what the source dataset has. The 15-minute run is a backup, not the normal way work arrives.

#### Can reviewers fix an annotation or add classes?

No. The tool is read-only for this node. To get annotations fixed, send those assets to an outcome dataset that feeds a [Labeling](/node-library/nodes/label-labeling) node.

#### What happens when I delete the node?

Every asset still in a reviewer's queue goes back to the source dataset. The outcome datasets keep everything already judged.

#### Can one step review images and videos together?

No. The run fails when the source dataset and the reviewers' queues hold both images and videos.

## Inputs

**`Source dataset`** `dataset` — required

Dataset with the labeled assets to review. Assets a reviewer doesn't judge come back here. Key: `SOURCE_DATASET`.

---

**`Decision buttons`** `labeling_outcomes` — required, default: \[\{'id': 'accept', 'label': 'Accept', 'dataset': None}, \{'id': 'reject', 'label': 'Reject', 'dataset': None}]

Buttons shown in the annotation tool, each with a destination dataset. Clicking a button moves the current item there, with its annotations. Flow variables aren't accepted. Key: `OUTCOMES`.

---

**`Reviewers`** `users` — required

Workspace members who review. Each person gets a personal queue from the source dataset. Key: `LABELERS`.

---

**`Classes`** `ontology` — required

Classes and tags to review in. Defaults to the source dataset's ontology. You can pick another one or create one here. Tick some entries to show only those to reviewers, or tick nothing to show all of them. The queue still uses the whole ontology, so the annotations stay valid in it, and entries added to the ontology later are shown. Values come from `SOURCE_DATASET`. Key: `CLASSES`.

---

**`Queue order`** `select` — required, default: arrival

Which assets go into a reviewer's queue first, and the order the tool shows them in. Highest anomaly score first reads the score Embedding anomalies saves to each image, so the hardest cases come first. Images without a score follow, oldest first. Flow variables aren't accepted. Key: `QUEUE_ORDER`.

Options:

* Oldest first (`arrival`)
* Highest anomaly score first (`anomaly_score`)

---

**`Batch size`** `integer` — required, default: 10

How many assets a reviewer holds at once. The queue refills to this number after each verdict and on every node run. Minimum: `1`. Key: `BATCH_SIZE`.

---

**`Return unclaimed work after (minutes)`** `integer` — required, default: 120

Assets that stay in a reviewer's queue longer than this without a verdict go back to the source dataset. This keeps unfinished work from getting stuck. Minimum: `1`. Key: `CLAIM_TTL_MINUTES`.

---

## Outputs

**`User queues`** `object`

Internal map of user IDs to their hidden review queue datasets. Hidden from the flow editor. Key: `USERS_DATASETS`.

---

**`Outcome datasets`** `object`

Internal list of allowed outcome dataset IDs and button labels. Hidden from the flow editor. Key: `OUTCOME_DATASETS`.

---

**`Status`** `object`

Compact per-reviewer queue status table object. Hidden from the flow editor. Key: `STATUS`.

---

## Widgets

The node card shows a live chart of the work: assets in progress in reviewers' queues and how many assets they accepted or rejected. Below the chart is a list of reviewers with the number of items each one reviewed. Click a reviewer to see only their numbers. The chart updates a few seconds after every **Accept** or **Reject**, without a node run.

**Open annotation tool** opens your own queue. It works only if you're in **Reviewers**, and only after the node has run once since you were added. Reviewers can also open their queue from the **Labeling** page. The tool opens the image or video interface to match what the queue holds.

## JSON config

Machine-readable node interface for automation and advanced usage.

#### Show JSON

```json
{
  "name": "Review",
  "description": "People check labeled images or videos here. They can't change the labels. A button, such as Accept or Reject, sends the item to its dataset.",
  "category": "Label",
  "namespace": "ovalbee",
  "templateKey": "label/review",
  "version": "v2",
  "inputs": [
    {
      "key": "SOURCE_DATASET",
      "label": "Source dataset",
      "type": "dataset",
      "description": "Dataset with the labeled assets to review. Assets a reviewer doesn't judge come back here.",
      "required": true,
      "default": null,
      "visibleWhen": null,
      "options": {
        "creatable": false
      }
    },
    {
      "key": "OUTCOMES",
      "label": "Decision buttons",
      "type": "labeling_outcomes",
      "description": "Buttons shown in the annotation tool, each with a destination dataset. Clicking a button moves the current item there, with its annotations.",
      "required": true,
      "default": [
        {
          "id": "accept",
          "label": "Accept",
          "dataset": null
        },
        {
          "id": "reject",
          "label": "Reject",
          "dataset": null
        }
      ],
      "visibleWhen": null,
      "options": {
        "allowVariables": false
      }
    },
    {
      "key": "LABELERS",
      "label": "Reviewers",
      "type": "users",
      "description": "Workspace members who review. Each person gets a personal queue from the source dataset.",
      "required": true,
      "default": null,
      "visibleWhen": null
    },
    {
      "key": "CLASSES",
      "label": "Classes",
      "type": "ontology",
      "description": "Classes and tags to review in. Defaults to the source dataset's ontology. You can pick another one or create one here. Tick some entries to show only those to reviewers, or tick nothing to show all of them. The queue still uses the whole ontology, so the annotations stay valid in it, and entries added to the ontology later are shown.",
      "required": true,
      "default": null,
      "visibleWhen": null,
      "options": {
        "ref": "SOURCE_DATASET",
        "selectable": true,
        "creatable": true
      }
    },
    {
      "key": "QUEUE_ORDER",
      "label": "Queue order",
      "type": "select",
      "description": "Which assets go into a reviewer's queue first, and the order the tool shows them in. Highest anomaly score first reads the score Embedding anomalies saves to each image, so the hardest cases come first. Images without a score follow, oldest first.",
      "required": true,
      "default": "arrival",
      "visibleWhen": null,
      "options": {
        "allowVariables": false,
        "options": [
          {
            "label": "Oldest first",
            "value": "arrival"
          },
          {
            "label": "Highest anomaly score first",
            "value": "anomaly_score"
          }
        ]
      }
    },
    {
      "key": "BATCH_SIZE",
      "label": "Batch size",
      "type": "number",
      "description": "How many assets a reviewer holds at once. The queue refills to this number after each verdict and on every node run.",
      "required": true,
      "default": 10,
      "visibleWhen": null,
      "options": {
        "type": "integer",
        "min": 1
      }
    },
    {
      "key": "CLAIM_TTL_MINUTES",
      "label": "Return unclaimed work after (minutes)",
      "type": "number",
      "description": "Assets that stay in a reviewer's queue longer than this without a verdict go back to the source dataset. This keeps unfinished work from getting stuck.",
      "required": true,
      "default": 120,
      "visibleWhen": null,
      "options": {
        "type": "integer",
        "min": 1
      }
    }
  ],
  "outputs": [
    {
      "key": "USERS_DATASETS",
      "label": "User queues",
      "type": "object",
      "description": "Internal map of user IDs to their hidden review queue datasets.",
      "artifact": false,
      "badge": false,
      "badgeOpens": null,
      "hidden": true
    },
    {
      "key": "OUTCOME_DATASETS",
      "label": "Outcome datasets",
      "type": "object",
      "description": "Internal list of allowed outcome dataset IDs and button labels.",
      "artifact": false,
      "badge": false,
      "badgeOpens": null,
      "hidden": true
    },
    {
      "key": "STATUS",
      "label": "Status",
      "type": "object",
      "description": "Compact per-reviewer queue status table object.",
      "artifact": false,
      "badge": false,
      "badgeOpens": null,
      "hidden": true
    }
  ],
  "automation": {
    "requires_configuration": true,
    "schedule": "*/15 * * * *",
    "discard_history": "on_success"
  },
  "widgets": {
    "widget": {
      "id": "labeling-stats",
      "settings": {}
    },
    "secondaryWidget": {
      "id": "open-labeling-tool",
      "settings": {
        "name": {
          "type": "input",
          "value": "Open annotation tool"
        }
      }
    }
  }
}
```