Labeling queue manager

Overview

Runs one Supervisely labeling queue for a whole flow step. Supervisely hands entities out and owns the decisions people make, while this node keeps the pool topped up, adds people who join, and mirrors the result onto six OvalBee datasets that show where every image stands.

Accepted images arrive back in OvalBee with the annotations people drew, ready for the rest of the flow. The card itself is one button into Supervisely — the numbers live in the stage datasets and in the Labeling stats and Review stats nodes. Once every required input is set, the node runs itself every minute.

Prerequisites

Requirement	What you need
Supervisely credentials	An existing `supervisely` secret allowed to manage labeling queues in the chosen team.
Supervisely project (optional)	An existing project, when the step should work in one. Without it the node creates its own. The node creates its own dataset inside it and reuses that dataset for every stage and rework round.
Team membership	Every labeler and reviewer already a member of the team that owns the project.

How it works

Uploads source images into one dataset of the chosen project, once each, and adds them to the queue pool.

Creates the queue over that pool on the first run and reuses it afterwards, adding labelers and reviewers who appear in the inputs later.

Reads the pool and the review jobs, then moves each image into the stage dataset that matches its position.

Pulls the current annotation back into OvalBee for images that reached review, rework or accepted.

Reopens a labeler’s finished job when new images arrive, as long as nothing in it was submitted.

FAQ

Where do the stage datasets come from?

The node creates and owns five hidden datasets for the stages in progress and one result dataset for accepted work. Source images are never copied. Each image leaves the source dataset when it enters the first stage, and deleting the node returns the images still in progress. Accepted images stay where they are.

What happens when the queue is stopped or completed in Supervisely?

The node stops touching Supervisely and keeps the stored identifiers. Each run still completes, writes the reason to its log, and shows a badge on the card. It never replaces a finished queue with a fresh one behind your back.

Why did the run fail before anything happened in Supervisely?

The step needs at least one labeler, at least one reviewer, because the queue always has a review stage, and a non-empty Classes vocabulary.

Why does somebody still hold work after I removed them from the inputs?

Supervisely has no way to take a person out of a live queue. Their job stays, and the node reports it as a note rather than rebuilding the queue and losing everybody’s statuses.

Does an image stay in Rework after its author resubmits it?

No. Rework counts rejected images whose author has not resubmitted them. As soon as the work comes back, the image moves on to Waiting for review.

Does the node ever submit or finish work for people?

No. Labelers submit and reviewers decide in Supervisely. The only job status the node changes is reopening a labeling job that was finished before anything in it was submitted, and that is what the Reopen finished labeling jobs switch controls. Supervisely resets every image of a reopened job, so a job with submitted work stays finished and its labeler is out of the queue.

What do the badges on the card mean?

Each one repeats a warning from the run log and disappears once its reason does. Queue stopped, Queue completed, Queue finished or Queue missing means the queue was closed or removed in Supervisely and the node stopped touching it. N still assigned means people removed from the inputs who still hold a job in the queue. N not uploaded means source images that could not reach Supervisely yet and wait for the next run. N left the queue means labelers who finished their queue job with submitted work in it, which the node cannot reopen.

What does this node not support?

Batches, because a queue hands out one entity at a time. Use Labeling jobs manager when people need a named batch each. Classes added to the ontology later are added to the Supervisely project on the next run, but nothing is ever taken out of it, so removing a class needs a new step. Quality check is out of scope for this version, and rejected rounds leave no historical snapshot behind, only the current annotation.

Inputs

Source dataset

datasetRequired

Images to label. Each image leaves this dataset when it enters the first stage. Deleting the node returns the images still in progress. Key: SOURCE_DATASET.

Supervisely credentials

secretRequired

Allowed credential types: supervisely. Key: SLY_SECRET.

Supervisely team

supervisely_selectRequired

Team the work happens in. It also scopes the labeler and reviewer lists. Values come from SLY_SECRET. Key: SLY_TEAM.

Supervisely workspace

supervisely_select

Workspace that holds the project. Empty means an OvalBee workspace in the team, used only when the project is created for you. Values come from SLY_SECRET. Key: SLY_WORKSPACE.

Supervisely project

supervisely_select

Project the whole step works in. Images are uploaded into one dataset of this project and reused by every stage and rework round. Empty means this node creates its own project and reuses it on every run. Another project can be chosen only until the queue exists. Values come from SLY_SECRET. Key: SLY_PROJECT.

Labelers

supervisely_selectRequired

People who label the images. Values come from SLY_SECRET. Key: LABELERS.

Reviewers

supervisely_selectRequired

People who review what the labelers submit. Values come from SLY_SECRET. Key: REVIEWERS.

Classes

ontologyRequired

Vocabulary this step labels in. Defaults to the source dataset’s ontology. Tick part of it to show labelers only those classes, or tick nothing to label in all of it. Values come from SOURCE_DATASET. Key: CLASSES.

Accepted dataset

dataset

Where accepted images and their annotations land. Empty means this node creates and reuses its own result dataset. Key: ACCEPTED_DATASET.

Reviewers may review their own labels

booleanDefaults to falseRequired

Key: ALLOW_REVIEW_OWN.

Hide figure author

booleanDefaults to falseRequired

Hide who drew each figure from the people working the queue. Key: HIDE_FIGURE_AUTHOR.

Reopen finished labeling jobs

booleanDefaults to trueRequired

A labeler may finish their job whenever the pool runs dry. Keep this on to put them back to work when new images arrive. Only a job with nothing submitted in it can be reopened. Key: REVIVE_COMPLETED_JOBS.

Outputs

Waiting for labeling

dataset

Images in the pool that Supervisely has not handed to anybody yet. Key: WAITING.

Labeling

dataset

Images handed to a labeler and not submitted yet. Key: LABELING.

Waiting for review

dataset

Submitted images no reviewer holds yet. Key: TO_REVIEW.

Review

dataset

Images a reviewer holds without a decision. Key: REVIEWING.

Rework

dataset

Rejected images their author has not resubmitted yet. Key: REWORK.

Accepted

dataset

Accepted images with the annotations pulled back into OvalBee. Shown as an artifact. Key: ACCEPTED.

Queue issue

string

Badge shown when the queue was stopped, finished or removed in Supervisely. Shown on the node as a badge. Hidden from the flow editor. Key: QUEUE_ISSUE.

Still assigned

string

Badge counting people removed from the inputs who still hold a queue job. Shown on the node as a badge. Hidden from the flow editor. Key: STILL_ASSIGNED.

Not uploaded

string

Badge counting source images that could not reach Supervisely yet. Shown on the node as a badge. Hidden from the flow editor. Key: NOT_UPLOADED.

Left the queue

string

Badge counting labelers who finished their queue job with submitted work in it, which cannot be reopened without resetting that work. Shown on the node as a badge. Hidden from the flow editor. Key: LEFT_QUEUE.

Process

object

Handle of this labeling process. Connect it to the Labeling stats and Review stats nodes. Key: PROCESS.

Supervisely queue URL

string

Link to the Supervisely labeling queues list. Key: SLY_QUEUE_URL.

State

object

Internal map of the Supervisely resources this node owns. Hidden from the flow editor. Key: STATE.

JSON config

Machine-readable node interface for automation and advanced usage.

Show JSON

{
  "name": "Labeling queue manager",
  "description": "Runs one Supervisely labeling queue for a whole step and shows where every image stands as OvalBee stage datasets.",
  "category": "Supervisely",
  "namespace": "supervisely",
  "templateKey": "supervisely/labeling_queue_manager",
  "version": "v1",
  "inputs": [
    {
      "key": "SOURCE_DATASET",
      "label": "Source dataset",
      "type": "dataset",
      "description": "Images to label. Each image leaves this dataset when it enters the first stage. Deleting the node returns the images still in progress.",
      "required": true,
      "default": null,
      "visibleWhen": null
    },
    {
      "key": "SLY_SECRET",
      "label": "Supervisely credentials",
      "type": "secret",
      "description": "",
      "required": true,
      "default": null,
      "visibleWhen": null,
      "options": {
        "types": [
          "supervisely"
        ]
      }
    },
    {
      "key": "SLY_TEAM",
      "label": "Supervisely team",
      "type": "supervisely_select",
      "description": "Team the work happens in. It also scopes the labeler and reviewer lists.",
      "required": true,
      "default": null,
      "visibleWhen": null,
      "options": {
        "kind": "team",
        "ref": "SLY_SECRET"
      }
    },
    {
      "key": "SLY_WORKSPACE",
      "label": "Supervisely workspace",
      "type": "supervisely_select",
      "description": "Workspace that holds the project. Empty means an `OvalBee` workspace in the team, used only when the project is created for you.",
      "required": false,
      "default": null,
      "visibleWhen": null,
      "options": {
        "kind": "workspace",
        "ref": "SLY_SECRET",
        "parent_ref": "SLY_TEAM"
      }
    },
    {
      "key": "SLY_PROJECT",
      "label": "Supervisely project",
      "type": "supervisely_select",
      "description": "Project the whole step works in. Images are uploaded into one dataset of this project and reused by every stage and rework round. Empty means this node creates its own project and reuses it on every run. Another project can be chosen only until the queue exists.",
      "required": false,
      "default": null,
      "visibleWhen": null,
      "options": {
        "kind": "project",
        "ref": "SLY_SECRET",
        "parent_ref": "SLY_WORKSPACE"
      }
    },
    {
      "key": "LABELERS",
      "label": "Labelers",
      "type": "supervisely_select",
      "description": "People who label the images.",
      "required": true,
      "default": null,
      "visibleWhen": null,
      "options": {
        "kind": "member",
        "multiple": true,
        "ref": "SLY_SECRET",
        "parent_ref": "SLY_TEAM"
      }
    },
    {
      "key": "REVIEWERS",
      "label": "Reviewers",
      "type": "supervisely_select",
      "description": "People who review what the labelers submit.",
      "required": true,
      "default": null,
      "visibleWhen": null,
      "options": {
        "kind": "member",
        "multiple": true,
        "ref": "SLY_SECRET",
        "parent_ref": "SLY_TEAM"
      }
    },
    {
      "key": "CLASSES",
      "label": "Classes",
      "type": "ontology",
      "description": "Vocabulary this step labels in. Defaults to the source dataset's ontology. Tick part of it to show labelers only those classes, or tick nothing to label in all of it.",
      "required": true,
      "default": null,
      "visibleWhen": null,
      "options": {
        "ref": "SOURCE_DATASET",
        "selectable": true,
        "creatable": true
      }
    },
    {
      "key": "ACCEPTED_DATASET",
      "label": "Accepted dataset",
      "type": "dataset",
      "description": "Where accepted images and their annotations land. Empty means this node creates and reuses its own result dataset.",
      "required": false,
      "default": null,
      "visibleWhen": null
    },
    {
      "key": "ALLOW_REVIEW_OWN",
      "label": "Reviewers may review their own labels",
      "type": "boolean",
      "description": "",
      "required": true,
      "default": false,
      "visibleWhen": null
    },
    {
      "key": "HIDE_FIGURE_AUTHOR",
      "label": "Hide figure author",
      "type": "boolean",
      "description": "Hide who drew each figure from the people working the queue.",
      "required": true,
      "default": false,
      "visibleWhen": null
    },
    {
      "key": "REVIVE_COMPLETED_JOBS",
      "label": "Reopen finished labeling jobs",
      "type": "boolean",
      "description": "A labeler may finish their job whenever the pool runs dry. Keep this on to put them back to work when new images arrive. Only a job with nothing submitted in it can be reopened.",
      "required": true,
      "default": true,
      "visibleWhen": null
    }
  ],
  "outputs": [
    {
      "key": "WAITING",
      "label": "Waiting for labeling",
      "type": "dataset",
      "description": "Images in the pool that Supervisely has not handed to anybody yet.",
      "artifact": false,
      "badge": false,
      "badgeOpens": null,
      "hidden": false
    },
    {
      "key": "LABELING",
      "label": "Labeling",
      "type": "dataset",
      "description": "Images handed to a labeler and not submitted yet.",
      "artifact": false,
      "badge": false,
      "badgeOpens": null,
      "hidden": false
    },
    {
      "key": "TO_REVIEW",
      "label": "Waiting for review",
      "type": "dataset",
      "description": "Submitted images no reviewer holds yet.",
      "artifact": false,
      "badge": false,
      "badgeOpens": null,
      "hidden": false
    },
    {
      "key": "REVIEWING",
      "label": "Review",
      "type": "dataset",
      "description": "Images a reviewer holds without a decision.",
      "artifact": false,
      "badge": false,
      "badgeOpens": null,
      "hidden": false
    },
    {
      "key": "REWORK",
      "label": "Rework",
      "type": "dataset",
      "description": "Rejected images their author has not resubmitted yet.",
      "artifact": false,
      "badge": false,
      "badgeOpens": null,
      "hidden": false
    },
    {
      "key": "ACCEPTED",
      "label": "Accepted",
      "type": "dataset",
      "description": "Accepted images with the annotations pulled back into OvalBee.",
      "artifact": true,
      "badge": false,
      "badgeOpens": null,
      "hidden": false
    },
    {
      "key": "QUEUE_ISSUE",
      "label": "Queue issue",
      "type": "string",
      "description": "Badge shown when the queue was stopped, finished or removed in Supervisely.",
      "artifact": false,
      "badge": true,
      "badgeOpens": null,
      "hidden": true
    },
    {
      "key": "STILL_ASSIGNED",
      "label": "Still assigned",
      "type": "string",
      "description": "Badge counting people removed from the inputs who still hold a queue job.",
      "artifact": false,
      "badge": true,
      "badgeOpens": null,
      "hidden": true
    },
    {
      "key": "NOT_UPLOADED",
      "label": "Not uploaded",
      "type": "string",
      "description": "Badge counting source images that could not reach Supervisely yet.",
      "artifact": false,
      "badge": true,
      "badgeOpens": null,
      "hidden": true
    },
    {
      "key": "LEFT_QUEUE",
      "label": "Left the queue",
      "type": "string",
      "description": "Badge counting labelers who finished their queue job with submitted work in it, which cannot be reopened without resetting that work.",
      "artifact": false,
      "badge": true,
      "badgeOpens": null,
      "hidden": true
    },
    {
      "key": "PROCESS",
      "label": "Process",
      "type": "object",
      "description": "Handle of this labeling process. Connect it to the Labeling stats and Review stats nodes.",
      "artifact": false,
      "badge": false,
      "badgeOpens": null,
      "hidden": false
    },
    {
      "key": "SLY_QUEUE_URL",
      "label": "Supervisely queue URL",
      "type": "string",
      "description": "Link to the Supervisely labeling queues list.",
      "artifact": false,
      "badge": false,
      "badgeOpens": null,
      "hidden": false
    },
    {
      "key": "STATE",
      "label": "State",
      "type": "object",
      "description": "Internal map of the Supervisely resources this node owns.",
      "artifact": false,
      "badge": false,
      "badgeOpens": null,
      "hidden": true
    }
  ],
  "automation": {
    "isolated": true,
    "requires_configuration": true,
    "schedule": "* * * * *",
    "discard_history": "on_success"
  },
  "widgets": {
    "widget": {
      "id": "link",
      "settings": {
        "name": {
          "type": "input",
          "value": "Open queues"
        },
        "link": {
          "type": "variable",
          "value": "self.outputs.SLY_QUEUE_URL"
        }
      }
    }
  }
}