Node LibrarySupervisely

Labeling jobs manager

Overview

Cuts the pool into batches and creates an ordinary Supervisely labeling job for each one, pinned to a person and reviewed by a person. Review happens inside that same job once the annotator submits it, so nobody works from a shared queue and everybody knows what their batch is.

Rejected images come back as a new job over exactly those images, and accepted images arrive in OvalBee with the annotations people drew. Six OvalBee datasets show where every image stands, and the card itself is one button into Supervisely. Once every required input is set, the node runs itself every minute.

Source dataset Labeling jobs manager Accepted Waiting · Labeling · Waiting forreview · Review · Rework

Prerequisites

RequirementWhat you need
Supervisely credentialsAn existing supervisely secret allowed to create labeling jobs in the chosen team.
Supervisely project (optional)An existing project, when the step should work in one. Without it the node creates its own. The node creates its own dataset inside it, which is what lets one person’s rework be a single job.
Team membershipEvery labeler and reviewer already a member of the team that owns the project.

How it works

1

Uploads source images into one dataset of the chosen project, once each, and treats every image outside a job as the pool.

2

Reads the node’s jobs and moves each image into the stage dataset that matches its latest job’s status and the reviewer’s decision.

3

Pulls the current annotation back into OvalBee for images that just reached review, rework or accepted.

4

Tops every labeler up to Minimum active jobs per person, taking batches of Batch size images from the pool, and starts each new job.

5

Re-issues rejected images as one new job per annotator and reports jobs that are late.

FAQ

The step needs at least one labeler, at least one reviewer, because every job is reviewed by one person, and a non-empty Classes vocabulary.

Supervisely splits a job by dataset, so a batch spread over several datasets would become several jobs. Everything this node uploads lives in one dataset of the project, which keeps a person’s rework to a single job no matter which batches the rejected images came from.

Supervisely only restarts a job that is still on review. A batch that reached the end of its life cannot be reopened, so the rejected images become a fresh job with the same settings and the closed one stays closed as a record.

The annotation tool turns a labeler away from a job that has not been started yet, and only the job page in Supervisely starts it. The node starts every job it creates, so the links on the stats cards open straight into the tool.

From the rejection until the labeler submits the rework job. The new job is created right away, but the images stay in Rework while it is being labeled, so rejected work can be looked at before it is redone.

They stay in Supervisely. The Open jobs button leads to the full list, and the Labeling stats and Review stats nodes keep the totals they contributed to.

No. It writes a log line for jobs that are fully labeled and waiting for their annotator to submit, and for jobs past the reminder or escalation deadline, but Submit and Finish stay with the people who own them.

A person is unavailable once they are no longer listed in Labelers. With Rework goes to set to the same annotator, When that person is unavailable decides between holding their rejected images until they are listed again and handing them to another listed labeler. Rework is reviewed by works the same way for reviewers, except that it always falls back to another listed reviewer.

Each one repeats a warning from the run log and disappears once its reason does. Both deadlines count from when the job was created. N overdue means jobs past the escalation deadline. N late means jobs past the reminder deadline that wait for their annotator to submit or for a review. N to submit means jobs fully labeled but not submitted yet. N not uploaded means source images that could not reach Supervisely yet and wait for the next run.

A shared pool people pull from one image at a time. Use Labeling queue manager for that. Quality check is out of scope for this version, and a rejected round leaves no historical snapshot behind, only the current annotation.

Inputs

Source dataset
datasetRequired

Images to label. Each image leaves this dataset when it enters the first stage. Deleting the node returns the images still in progress. Key: SOURCE_DATASET.

Supervisely credentials
secretRequired

Allowed credential types: supervisely. Key: SLY_SECRET.

Supervisely team
supervisely_selectRequired

Team the work happens in. It also scopes the labeler and reviewer lists. Values come from SLY_SECRET. Key: SLY_TEAM.

Supervisely workspace
supervisely_select

Workspace that holds the project. Empty means an OvalBee workspace in the team, used only when the project is created for you. Values come from SLY_SECRET. Key: SLY_WORKSPACE.

Supervisely project
supervisely_select

Project the whole step works in. Images are uploaded into one dataset of this project, which is what lets one person’s rework be a single job. Empty means this node creates its own project and reuses it on every run. Another project can be chosen only until the first job exists. Values come from SLY_SECRET. Key: SLY_PROJECT.

Labelers
supervisely_selectRequired

People who receive the batches. Values come from SLY_SECRET. Key: LABELERS.

Reviewers
supervisely_selectRequired

People who review the batches. One reviewer is assigned per job. Values come from SLY_SECRET. Key: REVIEWERS.

Classes
ontologyRequired

Vocabulary this step labels in. Defaults to the source dataset’s ontology. Tick part of it to show labelers only those classes, or tick nothing to label in all of it. Values come from SOURCE_DATASET. Key: CLASSES.

Batch size
integerDefaults to 50Required

How many images go into one job. Minimum: 1. Key: BATCH_SIZE.

Minimum active jobs per person
integerDefaults to 1Required

Keep topping a person up until they hold this many jobs they have not submitted. Minimum: 1. Key: MIN_ACTIVE_JOBS.

Re-issue rejected work
selectDefaults to autoRequired

On every scheduled run, or only when you press Run yourself. A rejected batch always becomes a new job over exactly the rejected images. Key: REWORK_MODE.

Options:

  • Automatically (auto)
  • Only on a manual run (manual)
Rework goes to
selectDefaults to sameRequired

Key: REWORK_ANNOTATOR.

Options:

  • The same annotator (same)
  • Any available annotator (any)
Rework is reviewed by
selectDefaults to sameRequired

Key: REWORK_REVIEWER.

Options:

  • The same reviewer (same)
  • Any available reviewer (any)
When that person is unavailable
selectDefaults to waitRequired

Wait for them to come back, or give the work to somebody who is configured. Visible when REWORK_ANNOTATOR is same. Key: ON_UNAVAILABLE.

Options:

  • Wait for them (wait)
  • Reassign (reassign)
Accepted dataset
dataset

Where accepted images and their annotations land. Empty means this node creates and reuses its own result dataset. Key: ACCEPTED_DATASET.

Remind after (minutes)
integerDefaults to 240Required

A job waiting longer than this is reported in the run log. Minimum: 1. Key: REMIND_AFTER_MINUTES.

Escalate after (minutes)
integerDefaults to 1440Required

A job waiting longer than this is reported in the run log as past its deadline. Minimum: 1. Key: ESCALATE_AFTER_MINUTES.

Outputs

Waiting for labeling
dataset

Images that are not in any job yet. Key: WAITING.

Labeling
dataset

Images in a job the annotator has not submitted yet. Key: LABELING.

Waiting for review
dataset

Submitted jobs the reviewer has not started deciding. Key: TO_REVIEW.

Review
dataset

Submitted jobs with decisions under way. Key: REVIEWING.

Rework
dataset

Rejected images, waiting for their new job or already in it. Key: REWORK.

Accepted
dataset

Accepted images with the annotations pulled back into OvalBee. Shown as an artifact. Key: ACCEPTED.

Overdue
string

Badge counting jobs past the escalation deadline. Shown on the node as a badge. Hidden from the flow editor. Key: OVERDUE.

Late
string

Badge counting jobs past the reminder deadline. Shown on the node as a badge. Hidden from the flow editor. Key: LATE.

To submit
string

Badge counting fully labeled jobs waiting for their annotator to submit. Shown on the node as a badge. Hidden from the flow editor. Key: TO_SUBMIT.

Not uploaded
string

Badge counting source images that could not reach Supervisely yet. Shown on the node as a badge. Hidden from the flow editor. Key: NOT_UPLOADED.

Process
object

Handle of this labeling process. Connect it to the Labeling stats and Review stats nodes. Key: PROCESS.

Supervisely jobs URL
string

Link to the Supervisely labeling jobs list. Key: SLY_JOBS_URL.

State
object

Internal map of the Supervisely resources this node owns. Hidden from the flow editor. Key: STATE.

JSON config

Machine-readable node interface for automation and advanced usage.

{
  "name": "Labeling jobs manager",
  "description": "Hands batches out as ordinary Supervisely jobs, one batch per person, and re-issues rejected work as a new job.",
  "category": "Supervisely",
  "namespace": "supervisely",
  "templateKey": "supervisely/labeling_jobs_manager",
  "version": "v1",
  "inputs": [
    {
      "key": "SOURCE_DATASET",
      "label": "Source dataset",
      "type": "dataset",
      "description": "Images to label. Each image leaves this dataset when it enters the first stage. Deleting the node returns the images still in progress.",
      "required": true,
      "default": null,
      "visibleWhen": null
    },
    {
      "key": "SLY_SECRET",
      "label": "Supervisely credentials",
      "type": "secret",
      "description": "",
      "required": true,
      "default": null,
      "visibleWhen": null,
      "options": {
        "types": [
          "supervisely"
        ]
      }
    },
    {
      "key": "SLY_TEAM",
      "label": "Supervisely team",
      "type": "supervisely_select",
      "description": "Team the work happens in. It also scopes the labeler and reviewer lists.",
      "required": true,
      "default": null,
      "visibleWhen": null,
      "options": {
        "kind": "team",
        "ref": "SLY_SECRET"
      }
    },
    {
      "key": "SLY_WORKSPACE",
      "label": "Supervisely workspace",
      "type": "supervisely_select",
      "description": "Workspace that holds the project. Empty means an `OvalBee` workspace in the team, used only when the project is created for you.",
      "required": false,
      "default": null,
      "visibleWhen": null,
      "options": {
        "kind": "workspace",
        "ref": "SLY_SECRET",
        "parent_ref": "SLY_TEAM"
      }
    },
    {
      "key": "SLY_PROJECT",
      "label": "Supervisely project",
      "type": "supervisely_select",
      "description": "Project the whole step works in. Images are uploaded into one dataset of this project, which is what lets one person's rework be a single job. Empty means this node creates its own project and reuses it on every run. Another project can be chosen only until the first job exists.",
      "required": false,
      "default": null,
      "visibleWhen": null,
      "options": {
        "kind": "project",
        "ref": "SLY_SECRET",
        "parent_ref": "SLY_WORKSPACE"
      }
    },
    {
      "key": "LABELERS",
      "label": "Labelers",
      "type": "supervisely_select",
      "description": "People who receive the batches.",
      "required": true,
      "default": null,
      "visibleWhen": null,
      "options": {
        "kind": "member",
        "multiple": true,
        "ref": "SLY_SECRET",
        "parent_ref": "SLY_TEAM"
      }
    },
    {
      "key": "REVIEWERS",
      "label": "Reviewers",
      "type": "supervisely_select",
      "description": "People who review the batches. One reviewer is assigned per job.",
      "required": true,
      "default": null,
      "visibleWhen": null,
      "options": {
        "kind": "member",
        "multiple": true,
        "ref": "SLY_SECRET",
        "parent_ref": "SLY_TEAM"
      }
    },
    {
      "key": "CLASSES",
      "label": "Classes",
      "type": "ontology",
      "description": "Vocabulary this step labels in. Defaults to the source dataset's ontology. Tick part of it to show labelers only those classes, or tick nothing to label in all of it.",
      "required": true,
      "default": null,
      "visibleWhen": null,
      "options": {
        "ref": "SOURCE_DATASET",
        "selectable": true,
        "creatable": true
      }
    },
    {
      "key": "BATCH_SIZE",
      "label": "Batch size",
      "type": "number",
      "description": "How many images go into one job.",
      "required": true,
      "default": 50,
      "visibleWhen": null,
      "options": {
        "type": "integer",
        "min": 1
      }
    },
    {
      "key": "MIN_ACTIVE_JOBS",
      "label": "Minimum active jobs per person",
      "type": "number",
      "description": "Keep topping a person up until they hold this many jobs they have not submitted.",
      "required": true,
      "default": 1,
      "visibleWhen": null,
      "options": {
        "type": "integer",
        "min": 1
      }
    },
    {
      "key": "REWORK_MODE",
      "label": "Re-issue rejected work",
      "type": "select",
      "description": "On every scheduled run, or only when you press Run yourself. A rejected batch always becomes a new job over exactly the rejected images.",
      "required": true,
      "default": "auto",
      "visibleWhen": null,
      "options": {
        "options": [
          {
            "label": "Automatically",
            "value": "auto"
          },
          {
            "label": "Only on a manual run",
            "value": "manual"
          }
        ]
      }
    },
    {
      "key": "REWORK_ANNOTATOR",
      "label": "Rework goes to",
      "type": "select",
      "description": "",
      "required": true,
      "default": "same",
      "visibleWhen": null,
      "options": {
        "options": [
          {
            "label": "The same annotator",
            "value": "same"
          },
          {
            "label": "Any available annotator",
            "value": "any"
          }
        ]
      }
    },
    {
      "key": "REWORK_REVIEWER",
      "label": "Rework is reviewed by",
      "type": "select",
      "description": "",
      "required": true,
      "default": "same",
      "visibleWhen": null,
      "options": {
        "options": [
          {
            "label": "The same reviewer",
            "value": "same"
          },
          {
            "label": "Any available reviewer",
            "value": "any"
          }
        ]
      }
    },
    {
      "key": "ON_UNAVAILABLE",
      "label": "When that person is unavailable",
      "type": "select",
      "description": "Wait for them to come back, or give the work to somebody who is configured.",
      "required": true,
      "default": "wait",
      "visibleWhen": {
        "key": "REWORK_ANNOTATOR",
        "operator": "equals",
        "value": "same"
      },
      "options": {
        "options": [
          {
            "label": "Wait for them",
            "value": "wait"
          },
          {
            "label": "Reassign",
            "value": "reassign"
          }
        ]
      }
    },
    {
      "key": "ACCEPTED_DATASET",
      "label": "Accepted dataset",
      "type": "dataset",
      "description": "Where accepted images and their annotations land. Empty means this node creates and reuses its own result dataset.",
      "required": false,
      "default": null,
      "visibleWhen": null
    },
    {
      "key": "REMIND_AFTER_MINUTES",
      "label": "Remind after (minutes)",
      "type": "number",
      "description": "A job waiting longer than this is reported in the run log.",
      "required": true,
      "default": 240,
      "visibleWhen": null,
      "options": {
        "type": "integer",
        "min": 1
      }
    },
    {
      "key": "ESCALATE_AFTER_MINUTES",
      "label": "Escalate after (minutes)",
      "type": "number",
      "description": "A job waiting longer than this is reported in the run log as past its deadline.",
      "required": true,
      "default": 1440,
      "visibleWhen": null,
      "options": {
        "type": "integer",
        "min": 1
      }
    }
  ],
  "outputs": [
    {
      "key": "WAITING",
      "label": "Waiting for labeling",
      "type": "dataset",
      "description": "Images that are not in any job yet.",
      "artifact": false,
      "badge": false,
      "badgeOpens": null,
      "hidden": false
    },
    {
      "key": "LABELING",
      "label": "Labeling",
      "type": "dataset",
      "description": "Images in a job the annotator has not submitted yet.",
      "artifact": false,
      "badge": false,
      "badgeOpens": null,
      "hidden": false
    },
    {
      "key": "TO_REVIEW",
      "label": "Waiting for review",
      "type": "dataset",
      "description": "Submitted jobs the reviewer has not started deciding.",
      "artifact": false,
      "badge": false,
      "badgeOpens": null,
      "hidden": false
    },
    {
      "key": "REVIEWING",
      "label": "Review",
      "type": "dataset",
      "description": "Submitted jobs with decisions under way.",
      "artifact": false,
      "badge": false,
      "badgeOpens": null,
      "hidden": false
    },
    {
      "key": "REWORK",
      "label": "Rework",
      "type": "dataset",
      "description": "Rejected images, waiting for their new job or already in it.",
      "artifact": false,
      "badge": false,
      "badgeOpens": null,
      "hidden": false
    },
    {
      "key": "ACCEPTED",
      "label": "Accepted",
      "type": "dataset",
      "description": "Accepted images with the annotations pulled back into OvalBee.",
      "artifact": true,
      "badge": false,
      "badgeOpens": null,
      "hidden": false
    },
    {
      "key": "OVERDUE",
      "label": "Overdue",
      "type": "string",
      "description": "Badge counting jobs past the escalation deadline.",
      "artifact": false,
      "badge": true,
      "badgeOpens": null,
      "hidden": true
    },
    {
      "key": "LATE",
      "label": "Late",
      "type": "string",
      "description": "Badge counting jobs past the reminder deadline.",
      "artifact": false,
      "badge": true,
      "badgeOpens": null,
      "hidden": true
    },
    {
      "key": "TO_SUBMIT",
      "label": "To submit",
      "type": "string",
      "description": "Badge counting fully labeled jobs waiting for their annotator to submit.",
      "artifact": false,
      "badge": true,
      "badgeOpens": null,
      "hidden": true
    },
    {
      "key": "NOT_UPLOADED",
      "label": "Not uploaded",
      "type": "string",
      "description": "Badge counting source images that could not reach Supervisely yet.",
      "artifact": false,
      "badge": true,
      "badgeOpens": null,
      "hidden": true
    },
    {
      "key": "PROCESS",
      "label": "Process",
      "type": "object",
      "description": "Handle of this labeling process. Connect it to the Labeling stats and Review stats nodes.",
      "artifact": false,
      "badge": false,
      "badgeOpens": null,
      "hidden": false
    },
    {
      "key": "SLY_JOBS_URL",
      "label": "Supervisely jobs URL",
      "type": "string",
      "description": "Link to the Supervisely labeling jobs list.",
      "artifact": false,
      "badge": false,
      "badgeOpens": null,
      "hidden": false
    },
    {
      "key": "STATE",
      "label": "State",
      "type": "object",
      "description": "Internal map of the Supervisely resources this node owns.",
      "artifact": false,
      "badge": false,
      "badgeOpens": null,
      "hidden": true
    }
  ],
  "automation": {
    "isolated": true,
    "requires_configuration": true,
    "schedule": "* * * * *",
    "discard_history": "on_success"
  },
  "widgets": {
    "widget": {
      "id": "link",
      "settings": {
        "name": {
          "type": "input",
          "value": "Open jobs"
        },
        "link": {
          "type": "variable",
          "value": "self.outputs.SLY_JOBS_URL"
        }
      }
    }
  }
}