Organize labeling

A labeling flow moves images through stages — labeling, review, maybe an expert or an automated check — and keeps rejected work cycling back until it’s right. OvalBee runs the whole thing as one flow: labelers and reviewers only see their own queue in the annotation tool, while whoever runs the project watches every stage on the canvas. If you haven’t yet, read Labeling for how pools, stages and outcomes fit together.

Pick a starting point

Template	Choose it when
Labeling with review loop	One labeling stage and one review stage. Rejected work returns for another pass. The right first flow for most teams.
Labeling with expert	Some images are hard enough to need a specialist. Labelers pass those on, and the expert’s work goes through the same review.
Labeling with auto quality check	You want obvious mistakes — empty submissions, tiny or duplicate boxes — caught before a reviewer spends time on them.
Supervisely queue labeling / Labeling jobs in Supervisely	Your team labels in Supervisely and should keep doing so. See below.

Labeling with review loop

Nodes: Dataset nodes for each pool, Labeling, Review and a Move node for the way back.

Point the pool at your images

The Unlabeled pool is the source of the Labeling stage. Fill it from an import, or point its Dataset node at a dataset you already have.

Choose the classes

In the Labeling stage’s Classes input, pick an ontology and, if you like, only some of its classes. The review stage and every outcome dataset use the same vocabulary. Turn on Labelers can add classes and tags if the class list is still being worked out.

Assign people and batch sizes

Add Labelers to the Labeling stage and Reviewers to the Review stage. Batch size is how many images one person holds at a time — it’s per person, not the total — and each stage has its own. Unfinished work returns to the pool after the time you set.

Decide the outcomes

Outcome statuses are the buttons in the annotation tool. Each one is a name and the dataset it sends the image to — Labeled and Skipped for the labeling stage, Accepted and Rejected for review. Type a new name in the dataset picker to create a destination on the spot.

Run the stages

Run the Labeling and Review nodes once to create everyone’s queues. From then on they refresh every 15 minutes on their own, and people open their work from Labeling in the main sidebar or with Open annotation tool on the node.

Rejected images wait in Rejected until the Return to labeling node moves them back to the pool — run it when you want rework to start, or give it a schedule. The edge drawn from Rejected to it is a visual edge, so the loop shows on the canvas without ever triggering itself.

See it running: live example · case study

Escalating to an expert

The expert version adds one outcome and one stage. Labelers get a third button, Needs expert, that sends the image to a Needs expert pool. A second Labeling stage, assigned to your specialist, takes from that pool. Its Labeled outcome goes to the same pool the regular labelers feed, so everything passes the same review.

A labeler can draw what they’re sure of and leave the rest to the expert, or hand an image over untouched. The specialist only ever sees the hard cases.

See it running: live example · case study

Automated quality checks before review

A Labeling gate sits between labeling and review and checks every submission against rules: no empty submissions, a minimum and maximum number of objects, a minimum box side, no duplicate boxes on the same object, boxes within a sensible share of the image. Submissions that pass go on to the reviewer. The rest land in a Failed QC dataset, and a Return to labeling node moves them back to the pool with their annotations, so the labeler fixes their own work instead of starting over.

The template also profiles the accepted dataset with Annotation quality, so the result comes with a report.

See it running: live example · case study

Watching progress

On the canvas. Pool nodes show how many images each stage holds, and stage nodes show who has how much in progress.
Labeling statistics. Open Labeling → Statistics for submitted work, backlog and time in the tool per person, a per-class breakdown and the history of every image. Everything exports to CSV.
Labeling page. Everyone sees the stages they’re assigned to, with the number of items waiting in their queue.

Labeling in Supervisely

When the team labels in Supervisely, the flow runs the Supervisely side and keeps OvalBee informed:

Supervisely queue labeling uses the Labeling queue manager: one shared Supervisely queue where annotators take the next image as soon as they’re free, and reviewers send rejects straight back to their author.
Labeling jobs in Supervisely uses the Labeling jobs manager: images are handed out as named batches, one Supervisely job per annotator, and rejected images are reissued as a new job.

In both, every stage — waiting for labeling, labeling, waiting for review, review, rework, accepted — is mirrored into an OvalBee dataset, and Labeling stats and Review stats show running totals on the canvas. The manager nodes run every minute, so new images join the process on their own. You need a Supervisely secret with your Supervisely address and API token.

See it running: queue labeling · jobs · case studies on queues and jobs

Make it yours

Feed training directly. Point a training flow’s dataset input at the Accepted dataset with @latest, and every retrain uses everything accepted so far.
Add outcomes freely. Blurry, Out of scope, Needs second opinion — every outcome is just another dataset, and you decide what happens to it next.
Start from predictions. Put a prediction node in front of the pool so labelers correct a model’s boxes instead of drawing from scratch, and measure how much that helps with Evaluate prelabeling efficiency.