Simulate labeling with live training

Overview

Labels a dataset that already has ground truth through a live training session, one image at a time, the way a labeler would in the labeling tool. For each image the session first guesses its boxes, then gets the ground truth annotation as if a labeler had submitted it, and trains for a while before the next image opens.

It lets you find out how fast a live training session learns your data before anyone labels by hand. Every pre-label the session made goes to the Prediction history output, tagged with the training iteration it came from. Wire that into Evaluate live training prelabeling, with the same dataset as its labeled images, to see how much of the work the pre-labels would have saved, and how that grew as the session learned. The node badge shows how many images were labeled, the iteration the session reached, and what ended the last wait.

Prerequisites

Requirement	What you need
Live training session	A running session, from the Serve live training with Grounding DINO node. A fresh one gives a fair measurement: images it has already trained on are pre-labeled from memory.
Ground truth annotations	Images in Labeled images that already carry their annotation in that dataset. Images without one are skipped.

How it works

Reads the images of Labeled images that carry an annotation, in a random order that is the same on every run when Shuffle images is on.

Waits until the session answers, which it does once its model is loaded.

For each image, asks the session for a pre-label with the weights it has at that moment and keeps it in a new version of the prediction history. Then submits the image with its ground truth annotation and lets the session train for Iterations per image, or less if it pauses because its loss stopped falling.

After the last image, waits until the session pauses or has trained Iterations after the last image more, so a node pre-labeling after this one uses the session at its best. The session keeps running afterwards.

FAQ

Why are the first images missing from the prediction history?

A session refuses to predict until it knows a class, which it learns from the first submitted image. Images it refuses are still submitted and trained on, only without a pre-label, so they aren’t in the history.

Why does the run fail saying the session never started training?

A session trains only once it holds a minimum number of labeled images with objects, set by initial_samples in its config. If the dataset has fewer annotated images than that, the session never trains. Label more images, or lower initial_samples in the session’s config.

What happens when the session already holds labeled images?

The run goes ahead and logs a warning: the session doesn’t start from scratch, so its pre-labels are better than a fresh session’s would be. Start a new session for a measurement you can compare.

What happens when I run it again?

Every run adds a new version of the prediction history with that run’s pre-labels. With Shuffle images on, the images come in the same order every time, so two runs on fresh sessions compare directly. Images the session trained on in an earlier run are pre-labeled from memory.

What fails the run?

A dataset with no annotated image, a session that doesn’t answer within 30 minutes, and a session whose training stops with an error.

Inputs

Live training session

deploymentRequired

Running live training session to label with. Start it fresh, or images it has already trained on are pre-labeled from memory. Key: DEPLOYMENT.

Labeled images

datasetRequired

Images to label, each with its annotation as what the labeler submits. Images with no annotation are skipped. Key: DATASET.

Shuffle images

booleanDefaults to trueRequired

Label the images in a random order, the same on every run, instead of the dataset’s own order. Keep it on when the dataset is sorted by scene or by class, or the first images teach the session only part of it. Key: SHUFFLE.

Iterations per image

integerDefaults to 15Required

Training iterations the session runs after each submitted image before the next one is opened - the time a labeler spends on an image, counted in training rather than in seconds. A session that pauses because its loss stopped falling moves on sooner. Minimum: 0. Maximum: 10000. Step: 5. Key: ITERATIONS_PER_IMAGE.

Iterations after the last image

integerDefaults to 500Required

After the last image, wait until the session pauses because its loss stopped falling, or has trained this many more iterations, so a node pre-labeling after this one uses the session at its best. The session keeps training afterwards either way. Minimum: 0. Maximum: 100000. Step: 50. Key: FINAL_ITERATIONS.

Outputs

Prediction history

dataset

Hidden dataset of every labeled image with the pre-label the session opened it with, tagged with the session and its training iteration. Wire it into Prediction history of Evaluate live training prelabeling, with Labeled images as its Labeled images. Every run adds a version. Key: PREDICTION_HISTORY.

Trained

string

How many images were labeled, the training iteration the session reached, and what ended the last wait. Shown on the node as a badge. Key: TRAINED.

JSON config

Machine-readable node interface for automation and advanced usage.

Show JSON

{
  "name": "Simulate labeling with live training",
  "description": "Work through a labeled dataset the way a labeler does with a live training session - open each image with the session's pre-label, then submit its ground truth annotation - letting the session train between images. Use it to try live training on data that already has ground truth, and score its pre-labels with Evaluate live training prelabeling.",
  "category": "Label",
  "namespace": null,
  "templateKey": "models/general/simulate_labeling_with_live_training",
  "version": "v1",
  "inputs": [
    {
      "key": "DEPLOYMENT",
      "label": "Live training session",
      "type": "deployment",
      "description": "Running live training session to label with. Start it fresh, or images it has already trained on are pre-labeled from memory.",
      "required": true,
      "default": null,
      "visibleWhen": null,
      "options": {
        "kind": "live_train"
      }
    },
    {
      "key": "DATASET",
      "label": "Labeled images",
      "type": "dataset",
      "description": "Images to label, each with its annotation as what the labeler submits. Images with no annotation are skipped.",
      "required": true,
      "default": null,
      "visibleWhen": null,
      "options": {
        "creatable": false
      }
    },
    {
      "key": "SHUFFLE",
      "label": "Shuffle images",
      "type": "boolean",
      "description": "Label the images in a random order, the same on every run, instead of the dataset's own order. Keep it on when the dataset is sorted by scene or by class, or the first images teach the session only part of it.",
      "required": true,
      "default": true,
      "visibleWhen": null
    },
    {
      "key": "ITERATIONS_PER_IMAGE",
      "label": "Iterations per image",
      "type": "number",
      "description": "Training iterations the session runs after each submitted image before the next one is opened - the time a labeler spends on an image, counted in training rather than in seconds. A session that pauses because its loss stopped falling moves on sooner.",
      "required": true,
      "default": 15,
      "visibleWhen": null,
      "options": {
        "min": 0,
        "max": 10000,
        "step": 5,
        "type": "integer"
      }
    },
    {
      "key": "FINAL_ITERATIONS",
      "label": "Iterations after the last image",
      "type": "number",
      "description": "After the last image, wait until the session pauses because its loss stopped falling, or has trained this many more iterations, so a node pre-labeling after this one uses the session at its best. The session keeps training afterwards either way.",
      "required": true,
      "default": 500,
      "visibleWhen": null,
      "options": {
        "min": 0,
        "max": 100000,
        "step": 50,
        "type": "integer"
      }
    }
  ],
  "outputs": [
    {
      "key": "PREDICTION_HISTORY",
      "label": "Prediction history",
      "type": "dataset",
      "description": "Hidden dataset of every labeled image with the pre-label the session opened it with, tagged with the session and its training iteration. Wire it into Prediction history of Evaluate live training prelabeling, with Labeled images as its Labeled images. Every run adds a version.",
      "artifact": false,
      "badge": false,
      "badgeOpens": null,
      "hidden": false
    },
    {
      "key": "TRAINED",
      "label": "Trained",
      "type": "string",
      "description": "How many images were labeled, the training iteration the session reached, and what ended the last wait.",
      "artifact": false,
      "badge": true,
      "badgeOpens": null,
      "hidden": false
    }
  ]
}