Skip to navigation
Node libraryLabel

Labeling

Overview

Manages your labeling work, like a team lead. You give it a dataset of images or videos (assets) and a team of labelers, and it runs the work for you:

  • creates a small personal queue for each labeler, so nobody picks work by hand and two people never get the same asset;
  • refills a queue as soon as its labeler finishes an asset;
  • sends each finished asset to the right dataset, based on the button the labeler pressed;
  • takes back work that someone holds for too long, or that a removed labeler left behind.

You decide which buttons labelers see. In Decision buttons, each row is a button name and the dataset it sends assets to. A new node starts with two: Submit for finished work and Skip for assets the labeler sets aside. While an asset is in someone’s queue, it isn’t in the source dataset, so the source only holds work that nobody has taken yet.

Once every required input is set, the node runs by itself every 15 minutes to take back unfinished work. New assets in the source dataset don’t wait for that run: they go straight to queues with free space.

To check annotations that already exist without changing them, use Review.

Source dataset Labeler queue Outcome dataset of the clicked status

How it works

1

Creates a personal queue for each person in Labelers and fills it up to Batch size from the source dataset, in the order the assets arrived there, or highest anomaly score first when Queue order asks for it.

2

Moves an asset and its annotation to the button’s dataset as soon as the labeler presses a decision button in the annotation tool. The labeler’s queue refills right away.

3

Returns assets that stayed in a queue longer than Return unclaimed work after (minutes) to the source dataset.

4

Sends the queue of anyone you remove from Labelers back to the source dataset.

5

Adds the classes being labeled to each outcome dataset’s ontology. The Submit button’s dataset, or the only one when there’s a single button, becomes the Labeled dataset output, so later nodes can train on it or send more work to it.

FAQ

A button without a dataset gets a new one on the node’s first run. It’s named Labeled for Submit, Skipped for Skip, and after the button name for any other. The node then saves that dataset in the button’s row. You can also pick an existing dataset, or type a new name in the dataset picker to create one.

Each queue is tied to an ontology (the set of classes and tags), so every submission is saved against it. The run fails when Classes has no ontology. This happens when nothing is picked and the source dataset has no ontology, when class names are typed without an ontology, or when the picked ontology was deleted.

Every button needs a name. The run also fails when:

  • two buttons have the same name (letter case doesn’t matter);
  • two buttons use the same dataset;
  • there are no buttons at all;
  • a destination dataset no longer exists.

The buttons come from the node’s last run. After you change Decision buttons, run the node or wait for the next scheduled run. Then reload the tool.

Each run looks for classes being labeled that the outcome dataset’s ontology doesn’t have yet. These are the ticked classes, or all of them when nothing is ticked. The run adds them as a new ontology version and switches the dataset to it. An outcome dataset without an ontology gets the one picked in Classes. This way all outcome datasets share the same classes.

The timer starts when an asset enters a labeler’s queue, not at their last edit. The 15-minute run returns expired assets, so an asset can stay up to 15 minutes past the limit. Annotations the labeler saved but didn’t submit go back with the asset. The returned asset is placed after the assets already waiting in the source dataset.

A queue refills when its labeler submits an asset and when new assets arrive in the source dataset. It can only take what the source dataset has. The 15-minute run is a backup, not the normal way work arrives.

As soon as the labeler reloads the annotation tool. No node run is needed. New classes and tags are added as a new version to the ontology of the labeler’s queue and of each outcome dataset, and those datasets switch to it. The source dataset’s ontology doesn’t change.

Every asset still in a labeler’s queue goes back to the source dataset. The outcome datasets keep everything already submitted.

No. The run fails when the source dataset and the labelers’ queues hold both images and videos.

Inputs

Source dataset
datasetRequired

Dataset with the assets to label. Assets a labeler doesn’t submit come back here. Key: SOURCE_DATASET.

Decision buttons
labeling_outcomesDefaults to [{'id': 'submit', 'label': 'Submit', 'dataset': None}, {'id': 'skip', 'label': 'Skip', 'dataset': None}]Required

Buttons shown in the annotation tool, each with a destination dataset. Clicking a button moves the current item there, with its annotations. Flow variables aren’t accepted. Key: OUTCOMES.

Labelers
usersRequired

Workspace members who label. Each person gets a personal queue from the source dataset. Key: LABELERS.

Classes
ontologyRequired

Classes and tags to label with. Defaults to the source dataset’s ontology. You can pick another one or create one here. Tick some entries to show only those to labelers, or tick nothing to show all of them. The queue still uses the whole ontology, so the annotations stay valid in it, and entries added to the ontology later are shown. Values come from SOURCE_DATASET. Key: CLASSES.

Labelers can add classes and tags
booleanDefaults to falseRequired

Let labelers create classes and tags in the annotation tool. They’re added to the ontology as a new version, and this node’s queues and outcome datasets switch to it. Flow variables aren’t accepted. Key: ALLOW_NEW_CLASSES_AND_TAGS.

Queue order
selectDefaults to arrivalRequired

Which assets go into a labeler’s queue first, and the order the tool shows them in. Highest anomaly score first reads the score Embedding anomalies saves to each image, so the hardest cases come first. Images without a score follow, oldest first. Flow variables aren’t accepted. Key: QUEUE_ORDER.

Options:

  • Oldest first (arrival)
  • Highest anomaly score first (anomaly_score)
Batch size
integerDefaults to 10Required

How many assets a labeler holds at once. The queue refills to this number after each submit and on every node run. Minimum: 1. Key: BATCH_SIZE.

Return unclaimed work after (minutes)
integerDefaults to 120Required

Assets that stay in a labeler’s queue longer than this without being submitted go back to the source dataset. This keeps unfinished work from getting stuck. Minimum: 1. Key: CLAIM_TTL_MINUTES.

Outputs

User queues
object

Internal map of user IDs to their hidden labeling queue datasets. Hidden from the flow editor. Key: USERS_DATASETS.

Outcome datasets
object

Internal list of allowed outcome dataset IDs and button labels. Hidden from the flow editor. Key: OUTCOME_DATASETS.

Labeled dataset
dataset

Dataset where submitted assets land: the Submit button’s dataset, or the only one when there’s a single button. Empty when there are several buttons and none is Submit. Key: LABELED_DATASET.

Status
object

Compact per-labeler queue status table object. Hidden from the flow editor. Key: STATUS.

Widgets

The node card shows a live chart of the work: assets in progress in labelers’ queues, assets labeled and waiting for review, and assets a review accepted or rejected. Below the chart is a list of labelers with their acceptance rate and the number of items each one labeled. Click a labeler to see only their numbers. The chart updates a few seconds after every Submit, Accept or Reject, without a node run.

Open annotation tool opens your own queue. It works only if you’re in Labelers, and only after the node has run once since you were added. Labelers can also open their queue from the Labeling page. The tool opens the image or video interface to match what the queue holds.

JSON config

Machine-readable node interface for automation and advanced usage.

{
  "name": "Labeling",
  "description": "People label images or videos here. Each person gets a small set from the pool. A button in the tool, such as Submit or Skip, sends the item to its dataset.",
  "category": "Label",
  "namespace": "ovalbee",
  "templateKey": "label/labeling",
  "version": "v2",
  "inputs": [
    {
      "key": "SOURCE_DATASET",
      "label": "Source dataset",
      "type": "dataset",
      "description": "Dataset with the assets to label. Assets a labeler doesn't submit come back here.",
      "required": true,
      "default": null,
      "visibleWhen": null,
      "options": {
        "creatable": false
      }
    },
    {
      "key": "OUTCOMES",
      "label": "Decision buttons",
      "type": "labeling_outcomes",
      "description": "Buttons shown in the annotation tool, each with a destination dataset. Clicking a button moves the current item there, with its annotations.",
      "required": true,
      "default": [
        {
          "id": "submit",
          "label": "Submit",
          "dataset": null
        },
        {
          "id": "skip",
          "label": "Skip",
          "dataset": null
        }
      ],
      "visibleWhen": null,
      "options": {
        "allowVariables": false
      }
    },
    {
      "key": "LABELERS",
      "label": "Labelers",
      "type": "users",
      "description": "Workspace members who label. Each person gets a personal queue from the source dataset.",
      "required": true,
      "default": null,
      "visibleWhen": null
    },
    {
      "key": "CLASSES",
      "label": "Classes",
      "type": "ontology",
      "description": "Classes and tags to label with. Defaults to the source dataset's ontology. You can pick another one or create one here. Tick some entries to show only those to labelers, or tick nothing to show all of them. The queue still uses the whole ontology, so the annotations stay valid in it, and entries added to the ontology later are shown.",
      "required": true,
      "default": null,
      "visibleWhen": null,
      "options": {
        "ref": "SOURCE_DATASET",
        "selectable": true,
        "creatable": true
      }
    },
    {
      "key": "ALLOW_NEW_CLASSES_AND_TAGS",
      "label": "Labelers can add classes and tags",
      "type": "boolean",
      "description": "Let labelers create classes and tags in the annotation tool. They're added to the ontology as a new version, and this node's queues and outcome datasets switch to it.",
      "required": true,
      "default": false,
      "visibleWhen": null,
      "options": {
        "allowVariables": false
      }
    },
    {
      "key": "QUEUE_ORDER",
      "label": "Queue order",
      "type": "select",
      "description": "Which assets go into a labeler's queue first, and the order the tool shows them in. Highest anomaly score first reads the score Embedding anomalies saves to each image, so the hardest cases come first. Images without a score follow, oldest first.",
      "required": true,
      "default": "arrival",
      "visibleWhen": null,
      "options": {
        "allowVariables": false,
        "options": [
          {
            "label": "Oldest first",
            "value": "arrival"
          },
          {
            "label": "Highest anomaly score first",
            "value": "anomaly_score"
          }
        ]
      }
    },
    {
      "key": "BATCH_SIZE",
      "label": "Batch size",
      "type": "number",
      "description": "How many assets a labeler holds at once. The queue refills to this number after each submit and on every node run.",
      "required": true,
      "default": 10,
      "visibleWhen": null,
      "options": {
        "type": "integer",
        "min": 1
      }
    },
    {
      "key": "CLAIM_TTL_MINUTES",
      "label": "Return unclaimed work after (minutes)",
      "type": "number",
      "description": "Assets that stay in a labeler's queue longer than this without being submitted go back to the source dataset. This keeps unfinished work from getting stuck.",
      "required": true,
      "default": 120,
      "visibleWhen": null,
      "options": {
        "type": "integer",
        "min": 1
      }
    }
  ],
  "outputs": [
    {
      "key": "USERS_DATASETS",
      "label": "User queues",
      "type": "object",
      "description": "Internal map of user IDs to their hidden labeling queue datasets.",
      "artifact": false,
      "badge": false,
      "badgeOpens": null,
      "hidden": true
    },
    {
      "key": "OUTCOME_DATASETS",
      "label": "Outcome datasets",
      "type": "object",
      "description": "Internal list of allowed outcome dataset IDs and button labels.",
      "artifact": false,
      "badge": false,
      "badgeOpens": null,
      "hidden": true
    },
    {
      "key": "LABELED_DATASET",
      "label": "Labeled dataset",
      "type": "dataset",
      "description": "Dataset where submitted assets land: the Submit button's dataset, or the only one when there's a single button. Empty when there are several buttons and none is Submit.",
      "artifact": false,
      "badge": false,
      "badgeOpens": null,
      "hidden": false
    },
    {
      "key": "STATUS",
      "label": "Status",
      "type": "object",
      "description": "Compact per-labeler queue status table object.",
      "artifact": false,
      "badge": false,
      "badgeOpens": null,
      "hidden": true
    }
  ],
  "automation": {
    "requires_configuration": true,
    "schedule": "*/15 * * * *",
    "discard_history": "on_success"
  },
  "widgets": {
    "widget": {
      "id": "labeling-stats",
      "settings": {}
    },
    "secondaryWidget": {
      "id": "open-labeling-tool",
      "settings": {
        "name": {
          "type": "input",
          "value": "Open annotation tool"
        }
      }
    }
  }
}