Concepts

Labeling

Human labeling and review as stages of the flow, with every image's state visible on the canvas.

In OvalBee, labeling isn’t a separate project you export data to and import it back from. It’s part of the flow: people work in stages that are nodes, and the state of every image is the dataset it currently sits in. A labeling flow can be as simple as one labeler and one reviewer, or route hard cases to an expert, run automated quality checks before review, and send rejected work back for another pass.

Pools, stages and outcomes

A labeling flow is built from three things:

  • Pools are datasets holding work that’s waiting: Unlabeled, Labeled, Rejected.
  • Stages are Labeling and Review nodes. A stage takes items from its source pool and hands them to the people assigned to it.
  • Outcomes are the buttons people press in the annotation tool. Each outcome is a name paired with a destination dataset, and pressing the button moves the image and its annotation there.
Labeled Skipped Accepted Rejected Unlabeled Labeling Labeled Skipped Review Accepted Rejected Return to labeling

Because outcomes are just datasets you choose, the process is yours to design. Add a Needs expert outcome that feeds a second Labeling stage assigned to your specialist. Put a Labeling gate between labeling and review so rule-based checks send obviously broken annotations straight back. Let accepted images flow into a training dataset. The dashed edge in the diagram is a visual edge: the loop is drawn for the reader, and a Move node returns the rejected images to the pool.

How a stage hands out work

When a Labeling or Review node runs, it gives each assigned person a private queue filled from the source pool, up to the stage’s Batch size. The rest stays in the pool for others, so several people can work in parallel without claiming the same images.

  • Pressing an outcome button moves the current image on and refills the queue right away.
  • Work that sits in someone’s queue longer than the stage’s time limit goes back to the pool, so an abandoned batch doesn’t block the project.
  • Removing a person from the stage sends their queue back to the pool.
  • Stages refresh every 15 minutes on their own, so images that arrive in the pool later reach people without anyone rerunning the node.

The source pool only ever holds unclaimed work, and every image is in exactly one place. That’s what makes the counts on the canvas trustworthy.

Classes

Each stage has a Classes input: the ontology labelers work with, optionally narrowed to part of it. The outcome datasets are pinned to the same vocabulary, so labels stay consistent all the way down the flow.

Turn on Labelers can add classes and tags when the vocabulary is still being discovered. New entries are added to the ontology as a new version, and every later stage sees them.

The annotation tool

Labelers and reviewers open the annotation tool in a new browser tab, either with Open annotation tool on the node or from the Labeling page in the main sidebar, which lists every stage they’re assigned to. The tool shows their current batch, the class list, the drawing tools and the outcome buttons of that stage.

A few things to know:

  • Review stages open the tool read-only: reviewers judge annotations and route them with outcome buttons rather than editing them. A correction goes back through a labeling stage.
  • AI-assisted segmentation is available in the tool while an interactive segmentation model is deployed in the workspace.
  • A stage creates its queues on its first run. Until then, Open annotation tool tells you to run the node.

Keeping track

The node widget shows how much each person has in progress. For the full picture, open Labeling → Statistics (or the chart button next to a stage): submitted work, backlog, time spent in the tool, objects created and edited, outcomes per stage, a per-class breakdown and a per-image history. Every table exports to CSV.

Labeling in Supervisely

If your team already labels in Supervisely, keep it there. The Labeling queue manager and Labeling jobs manager nodes run Supervisely queues or jobs from a flow and mirror each stage — waiting, labeling, on review, rework, accepted — into OvalBee datasets, so the rest of the flow can build on the result.

Start from a template

The template library has ready labeling flows: Labeling with review loop, Labeling with expert, Labeling with auto quality check, Supervisely queue labeling and Labeling jobs in Supervisely. See Organize labeling for how to pick one and adapt it.