Skip to navigation
Node LibraryLabel

Consensus review

Overview

Shows every incoming image or video to every reviewer, who each route it to a configured status by pressing a button, and files the item under the status their votes decide. A new node starts with Accept and Reject. Each status either gets a new visible dataset on the first run, named Accepted for Accept, Rejected for Reject and after the status itself for any other, or routes to a dataset chosen in the flow editor. Reviewers never edit annotations: use Labeling for that, and Review for a step where the reviewers split the work and each item is judged once.

yes no Source dataset Awaiting consensus Reviewer 1 queue Reviewer 1 ballot Reviewer 2 queue Reviewer 2 ballot Reviewer 3 queue Reviewer 3 ballot Agreement reached? Winning status dataset Unresolved dataset

One reviewer is a legitimate setup, and is how an expert step settles what a larger panel could not.

How it works

An item’s position is dataset membership and nothing else, so a vote needs no state of its own. An item being voted on sits on a hidden pool that belongs to no reviewer and is a member of every reviewer’s queue at the same time. Counting the ballots that hold an item counts its votes.

1

Claims a batch from the source dataset onto the pool, and from there into every reviewer’s queue at once. The item belongs to several datasets and nothing is copied.

2

A reviewer pressing a status button casts a vote: the item leaves their queue for a hidden ballot recording the status they chose, while the pool still holds it and the others still see it.

3

Each scheduled run tallies the ballots. A status that satisfies the Agreement rule wins and the item moves there from the pool. If everyone has voted without a clear leader, or the item passed the time limit, it goes to the Unresolved dataset instead.

4

Either way the item is dropped from every queue and ballot, so a reviewer who has not voted stops seeing an item that is already decided.

Queue size is how many items one reviewer holds, and also the total taken from the source, because everyone is given the same items.

An item that passes the time limit goes to the Unresolved dataset, whether or not any votes were cast — returning it to the source would re-offer it to the same panel that already failed to settle it, and the votes cast so far are dropped with it.

The reviewer queues take the ontology of the source dataset, so each vote is recorded against it. A source dataset with no ontology still runs, and its votes are recorded without one.

Inputs

Source dataset
datasetRequired

Source dataset the assets put up for review are drawn from. Whatever the panel settles leaves it for the matching status dataset. Key: SOURCE_DATASET.

Outcome statuses
labeling_outcomesDefaults to [{'id': 'accept', 'label': 'Accept', 'dataset': None}, {'id': 'reject', 'label': 'Reject', 'dataset': None}]Required

The verdict buttons shown in the review tool, each with the dataset it sends assets to. A verdict left without a dataset gets a new one on the node’s first run. Flow variables aren’t accepted. Key: OUTCOMES.

Unresolved dataset
datasetRequired

Destination dataset for assets the panel could not settle - reviewers disagreed, or the time limit passed. Key: UNRESOLVED_DATASET.

Reviewers
usersRequired

Workspace members who review. Each sees every asset put up for review and votes on it. Key: REVIEWERS.

Agreement rule
selectDefaults to majorityRequired

Votes one status needs to win - more than half of the reviewers, or all of them. Key: AGREEMENT_RULE.

Options:

  • Majority (majority)
  • Unanimous (all)
Queue size
integerDefaults to 10Required

Number of assets up for review at once. Every reviewer sees all of them, and the next asset is drawn only after one of them is routed. Minimum: 1. Key: BATCH_SIZE.

Time limit per item (minutes)
integerDefaults to 120Required

How long an item may wait for a decision before it goes to the unresolved dataset. Minimum: 1. Key: CLAIM_TTL_MINUTES.

Outputs

Reviewer queues
object

Internal map of user IDs to their hidden review queue datasets. Hidden from the flow editor. Key: USERS_DATASETS.

Outcome datasets
object

Internal list of allowed status dataset IDs and button labels. Hidden from the flow editor. Key: OUTCOME_DATASETS.

Consensus datasets
object

Internal voting pool and per-reviewer ballot datasets. Hidden from the flow editor. Key: CONSENSUS_DATASETS.

Status
object

Compact per-reviewer status table object. Hidden from the flow editor. Key: STATUS.

JSON config

Machine-readable node interface for automation and advanced usage.

{
  "name": "Consensus review",
  "description": "Show every incoming item to every reviewer and route it to the status they agree on. An item the panel cannot settle goes to the unresolved dataset.",
  "category": "Label",
  "namespace": "ovalbee",
  "templateKey": "label/consensus_review",
  "version": "v2",
  "inputs": [
    {
      "key": "SOURCE_DATASET",
      "label": "Source dataset",
      "type": "dataset",
      "description": "Source dataset the assets put up for review are drawn from. Whatever the panel settles leaves it for the matching status dataset.",
      "required": true,
      "default": null,
      "visibleWhen": null,
      "options": {
        "creatable": false
      }
    },
    {
      "key": "OUTCOMES",
      "label": "Outcome statuses",
      "type": "labeling_outcomes",
      "description": "The verdict buttons shown in the review tool, each with the dataset it sends assets to. A verdict left without a dataset gets a new one on the node's first run.",
      "required": true,
      "default": [
        {
          "id": "accept",
          "label": "Accept",
          "dataset": null
        },
        {
          "id": "reject",
          "label": "Reject",
          "dataset": null
        }
      ],
      "visibleWhen": null,
      "options": {
        "allowVariables": false
      }
    },
    {
      "key": "UNRESOLVED_DATASET",
      "label": "Unresolved dataset",
      "type": "dataset",
      "description": "Destination dataset for assets the panel could not settle - reviewers disagreed, or the time limit passed.",
      "required": true,
      "default": null,
      "visibleWhen": null
    },
    {
      "key": "REVIEWERS",
      "label": "Reviewers",
      "type": "users",
      "description": "Workspace members who review. Each sees every asset put up for review and votes on it.",
      "required": true,
      "default": null,
      "visibleWhen": null
    },
    {
      "key": "AGREEMENT_RULE",
      "label": "Agreement rule",
      "type": "select",
      "description": "Votes one status needs to win - more than half of the reviewers, or all of them.",
      "required": true,
      "default": "majority",
      "visibleWhen": null,
      "options": {
        "options": [
          {
            "label": "Majority",
            "value": "majority"
          },
          {
            "label": "Unanimous",
            "value": "all"
          }
        ]
      }
    },
    {
      "key": "BATCH_SIZE",
      "label": "Queue size",
      "type": "number",
      "description": "Number of assets up for review at once. Every reviewer sees all of them, and the next asset is drawn only after one of them is routed.",
      "required": true,
      "default": 10,
      "visibleWhen": null,
      "options": {
        "type": "integer",
        "min": 1
      }
    },
    {
      "key": "CLAIM_TTL_MINUTES",
      "label": "Time limit per item (minutes)",
      "type": "number",
      "description": "How long an item may wait for a decision before it goes to the unresolved dataset.",
      "required": true,
      "default": 120,
      "visibleWhen": null,
      "options": {
        "type": "integer",
        "min": 1
      }
    }
  ],
  "outputs": [
    {
      "key": "USERS_DATASETS",
      "label": "Reviewer queues",
      "type": "object",
      "description": "Internal map of user IDs to their hidden review queue datasets.",
      "artifact": false,
      "badge": false,
      "badgeOpens": null,
      "hidden": true
    },
    {
      "key": "OUTCOME_DATASETS",
      "label": "Outcome datasets",
      "type": "object",
      "description": "Internal list of allowed status dataset IDs and button labels.",
      "artifact": false,
      "badge": false,
      "badgeOpens": null,
      "hidden": true
    },
    {
      "key": "CONSENSUS_DATASETS",
      "label": "Consensus datasets",
      "type": "object",
      "description": "Internal voting pool and per-reviewer ballot datasets.",
      "artifact": false,
      "badge": false,
      "badgeOpens": null,
      "hidden": true
    },
    {
      "key": "STATUS",
      "label": "Status",
      "type": "object",
      "description": "Compact per-reviewer status table object.",
      "artifact": false,
      "badge": false,
      "badgeOpens": null,
      "hidden": true
    }
  ],
  "automation": {
    "requires_configuration": true,
    "schedule": "*/15 * * * *",
    "discard_history": "on_success"
  },
  "widgets": {
    "widget": {
      "id": "table",
      "settings": {
        "data": {
          "type": "variable",
          "value": "self.outputs.STATUS"
        }
      }
    },
    "secondaryWidget": {
      "id": "open-labeling-tool",
      "settings": {
        "name": {
          "type": "input",
          "value": "Open review tool"
        }
      }
    }
  }
}