Node LibraryLabel

Labeling

Overview

Hands images or videos from a source dataset to labelers in small batches, and moves each asset a labeler submits to the dataset of the status button they clicked.

Each row in Outcome statuses pairs a button name with a destination dataset, for example Done with one dataset and Needs rework with another. In the annotation tool, clicking a button moves the asset and its annotation to that button’s dataset and hands the labeler the next asset. Pick an existing dataset for each status, or type a new name in the dataset picker to create one. Assets leave the source dataset while a labeler holds them, so the source only ever holds work nobody has claimed yet. For a read-only pass over annotations that are already there, use Review instead.

Once every required input is set, the node runs itself every 15 minutes to top up queues and take back unfinished work.

Source dataset Labeler queue Outcome dataset of the clicked status

How it works

1

Checks the source dataset and the Classes ontology, and adds the classes being labeled to each outcome dataset’s ontology.

2

Gives each person in Labelers a private queue, and sends the queue of anyone removed from the list back to the source dataset.

3

Returns assets that have sat in a queue longer than Return unclaimed work after (minutes) to the source dataset.

4

Tops each queue up to Batch size from the source dataset, in the order the assets arrived there.

5

Moves an asset and its annotation to the status’s dataset as soon as the labeler clicks a status button in the annotation tool, then refills that labeler’s queue right away.

FAQ

Every queue is pinned to an ontology, so each submission is recorded against it. The run fails when Classes names no ontology: nothing picked while the source dataset has none, class names written without an ontology, or an ontology deleted since it was picked.

Every status needs a name and a destination dataset. The run also fails when two statuses share a name (ignoring case), when two statuses point at the same dataset, when there are no statuses at all, or when a destination dataset no longer exists.

The buttons come from the node’s last run. After you change Outcome statuses, run the node or wait for the next scheduled run, then reload the tool.

Each run adds whichever of the classes being labeled (the ticked ones, or all of them) the outcome dataset’s ontology lacks, as a new version, and pins the dataset to it. An outcome dataset with no ontology gets a new ontology named after its status, holding those classes.

The clock starts when an asset enters a labeler’s queue, not at their last edit. Expired assets are returned on the node’s next run, so one can stay up to 15 minutes past the limit. An annotation the labeler saved but never submitted travels back with the asset, and the returned asset joins the end of the source dataset’s line.

A queue refills when its labeler submits and on each node run, and only from what the source dataset holds. Assets added to an empty source dataset reach waiting labelers at the next scheduled run, or right away if you run the node yourself.

Right after a reload of the annotation tool, with no node run needed. What labelers create is added as a new version to the ontology of their queue and of each outcome dataset, and those datasets move to it. The source dataset’s ontology is left as it is.

Every asset still in a labeler’s queue goes back to the source dataset. The outcome datasets keep what was already submitted.

No. The run fails when the source dataset and the labelers’ queues hold both images and videos.

Inputs

Source dataset
datasetRequired

Source dataset the assets to label are drawn from. Anything a labeler leaves unsubmitted comes back here. Key: SOURCE_DATASET.

Outcome statuses
labeling_outcomesRequired

Add the status buttons shown in the labeling tool and choose the destination dataset for each. Flow variables aren’t accepted. Key: OUTCOMES.

Labelers
usersRequired

Workspace members who label. Each gets a private queue filled from the source dataset. Key: LABELERS.

Classes
ontologyRequired

Vocabulary this step labels in. Defaults to the source dataset’s ontology. Pick another or create one here, and optionally tick only part of it - unticked entries are hidden from the labeler while the queue stays pinned to the whole vocabulary, so the annotations stay readable in it, and entries added to the ontology later are shown. Tick nothing to label in all of it. Values come from SOURCE_DATASET. Key: CLASSES.

Labelers can add classes and tags
booleanDefaults to falseRequired

Let labelers create classes and tags in the labeling tool. What they create is appended to the ontology as a new version, and this step’s queues and outcome datasets move to it. Flow variables aren’t accepted. Key: ALLOW_NEW_CLASSES_AND_TAGS.

Batch size
integerDefaults to 10Required

Number of assets a labeler holds at once. Their queue is topped up to this after each submit and on every node run. Minimum: 1. Key: BATCH_SIZE.

Return unclaimed work after (minutes)
integerDefaults to 120Required

Assets left unsubmitted in a labeler’s queue longer than this go back to the source dataset, so nobody sits on work they never finished. Minimum: 1. Key: CLAIM_TTL_MINUTES.

Outputs

User queues
object

Internal map of user IDs to their hidden labeling queue datasets. Hidden from the flow editor. Key: USERS_DATASETS.

Outcome datasets
object

Internal list of allowed outcome dataset IDs and button labels. Hidden from the flow editor. Key: OUTCOME_DATASETS.

Status
object

Compact per-labeler queue status table object. Hidden from the flow editor. Key: STATUS.

Widgets

The node shows a table with one row per labeler and the number of assets currently in their queue. Open annotation tool opens your own queue, so it works only for someone listed in Labelers, and only after the node has run once since they were added. Labelers can also open their queue from the Labeling page. The tool opens the image or video interface to match what the queue holds.

JSON config

Machine-readable node interface for automation and advanced usage.

{
  "name": "Labeling",
  "description": "Hand images or videos from a source dataset to labelers in small batches. Each outcome status becomes a button in the annotation tool that moves the submitted asset and its annotation to that status's dataset.",
  "category": "Label",
  "namespace": "ovalbee",
  "templateKey": "label/labeling",
  "version": "v2",
  "inputs": [
    {
      "key": "SOURCE_DATASET",
      "label": "Source dataset",
      "type": "dataset",
      "description": "Source dataset the assets to label are drawn from. Anything a labeler leaves unsubmitted comes back here.",
      "required": true,
      "default": null,
      "visibleWhen": null,
      "options": {
        "creatable": false
      }
    },
    {
      "key": "OUTCOMES",
      "label": "Outcome statuses",
      "type": "labeling_outcomes",
      "description": "Add the status buttons shown in the labeling tool and choose the destination dataset for each.",
      "required": true,
      "default": null,
      "visibleWhen": null,
      "options": {
        "allowVariables": false
      }
    },
    {
      "key": "LABELERS",
      "label": "Labelers",
      "type": "users",
      "description": "Workspace members who label. Each gets a private queue filled from the source dataset.",
      "required": true,
      "default": null,
      "visibleWhen": null
    },
    {
      "key": "CLASSES",
      "label": "Classes",
      "type": "ontology",
      "description": "Vocabulary this step labels in. Defaults to the source dataset's ontology. Pick another or create one here, and optionally tick only part of it - unticked entries are hidden from the labeler while the queue stays pinned to the whole vocabulary, so the annotations stay readable in it, and entries added to the ontology later are shown. Tick nothing to label in all of it.",
      "required": true,
      "default": null,
      "visibleWhen": null,
      "options": {
        "ref": "SOURCE_DATASET",
        "selectable": true,
        "creatable": true
      }
    },
    {
      "key": "ALLOW_NEW_CLASSES_AND_TAGS",
      "label": "Labelers can add classes and tags",
      "type": "boolean",
      "description": "Let labelers create classes and tags in the labeling tool. What they create is appended to the ontology as a new version, and this step's queues and outcome datasets move to it.",
      "required": true,
      "default": false,
      "visibleWhen": null,
      "options": {
        "allowVariables": false
      }
    },
    {
      "key": "BATCH_SIZE",
      "label": "Batch size",
      "type": "number",
      "description": "Number of assets a labeler holds at once. Their queue is topped up to this after each submit and on every node run.",
      "required": true,
      "default": 10,
      "visibleWhen": null,
      "options": {
        "type": "integer",
        "min": 1
      }
    },
    {
      "key": "CLAIM_TTL_MINUTES",
      "label": "Return unclaimed work after (minutes)",
      "type": "number",
      "description": "Assets left unsubmitted in a labeler's queue longer than this go back to the source dataset, so nobody sits on work they never finished.",
      "required": true,
      "default": 120,
      "visibleWhen": null,
      "options": {
        "type": "integer",
        "min": 1
      }
    }
  ],
  "outputs": [
    {
      "key": "USERS_DATASETS",
      "label": "User queues",
      "type": "object",
      "description": "Internal map of user IDs to their hidden labeling queue datasets.",
      "artifact": false,
      "badge": false,
      "badgeOpens": null,
      "hidden": true
    },
    {
      "key": "OUTCOME_DATASETS",
      "label": "Outcome datasets",
      "type": "object",
      "description": "Internal list of allowed outcome dataset IDs and button labels.",
      "artifact": false,
      "badge": false,
      "badgeOpens": null,
      "hidden": true
    },
    {
      "key": "STATUS",
      "label": "Status",
      "type": "object",
      "description": "Compact per-labeler queue status table object.",
      "artifact": false,
      "badge": false,
      "badgeOpens": null,
      "hidden": true
    }
  ],
  "automation": {
    "requires_configuration": true,
    "schedule": "*/15 * * * *",
    "discard_history": "on_success"
  },
  "widgets": {
    "widget": {
      "id": "table",
      "settings": {
        "data": {
          "type": "variable",
          "value": "self.outputs.STATUS"
        }
      }
    },
    "secondaryWidget": {
      "id": "open-labeling-tool",
      "settings": {
        "name": {
          "type": "input",
          "value": "Open annotation tool"
        }
      }
    }
  }
}