Skip to navigation
Node librarySample

Get next batch

Overview

Walks through a dataset in batches. Each run sends the next images, with their annotations, and an image sent once is never sent again by the same node. Put it after a dataset that keeps growing to feed the next steps a fixed amount of new data per run.

LAST_RUN opens the latest batch. A hidden SENT_ASSETS output points to the history dataset that keeps later runs from sending those images again. The badge shows the size of the latest batch and how many images were sent across all runs, such as 20 passed (5 new).

Dataset Get next batch Live training

How it works

1

Compares the images the source holds now with the node’s history of sent images.

2

Takes the first images not sent yet, up to the batch size, in the order they were added to the source.

3

Adds the images and their annotations to LAST_RUN and to the history dataset.

FAQ

Each run reads the source as it is at that moment. New images join the end of the queue and come out in the next batches, after the images that were already waiting.

The node stops the flow, returns no LAST_RUN dataset, and the badge shows no new. When new images arrive in the source, the next run sends them.

The node writes each batch into its own output dataset, named after the source, such as Next batch (Cats): created on the first run, a new empty version of it on every rerun, so LAST_RUN holds only the latest batch and the version history shows every earlier one.

The history belongs to this node and has no reset. A newly added Get next batch node starts from the first image.

No. This version sends image assets only.

Inputs

From dataset
datasetRequired

Dataset to walk through. It can keep growing, and each run reads its current contents. Key: SRC_DATASET.

Batch size
integerDefaults to 500Required

Number of images each run sends on. The last batch can be smaller. Minimum: 1. Step: 1. Key: BATCH_SIZE.

Outputs

Batch
dataset

Dataset with the images of the latest batch, named after the source dataset. A rerun writes a new version of it, and a new source starts a new dataset. Shown as an artifact. Key: LAST_RUN.

Progress
string

Short badge with the number of images in the latest batch and the number sent across all runs. Shown on the node as a badge. Key: PROGRESS.

Images in batch
string

Number of images in the latest batch, as a plain number to compare in a Condition. 0 when no unsent image was left. Key: BATCH_COUNT.

Sent Assets
dataset

ID of the dataset containing every image this node has sent across runs. Hidden from the flow editor. Key: SENT_ASSETS.

JSON config

Machine-readable node interface for automation and advanced usage.

{
  "name": "Get next batch",
  "description": "Walks through a dataset in batches. Each run takes the next images from the source, with their annotations, in the order they were added to it, and never returns an image twice. Images added to the source later are picked up by the next runs. When every image has been sent, the node stops the flow.",
  "category": "Sample",
  "namespace": "ovalbee",
  "templateKey": "sample/get_next_batch",
  "version": "v1",
  "inputs": [
    {
      "key": "SRC_DATASET",
      "label": "From dataset",
      "type": "dataset",
      "description": "Dataset to walk through. It can keep growing, and each run reads its current contents.",
      "required": true,
      "default": null,
      "visibleWhen": null,
      "options": {
        "creatable": false
      }
    },
    {
      "key": "BATCH_SIZE",
      "label": "Batch size",
      "type": "number",
      "description": "Number of images each run sends on. The last batch can be smaller.",
      "required": true,
      "default": 500,
      "visibleWhen": null,
      "options": {
        "type": "integer",
        "min": 1,
        "step": 1
      }
    }
  ],
  "outputs": [
    {
      "key": "LAST_RUN",
      "label": "Batch",
      "type": "dataset",
      "description": "Dataset with the images of the latest batch, named after the source dataset. A rerun writes a new version of it, and a new source starts a new dataset.",
      "artifact": true,
      "badge": false,
      "badgeOpens": null,
      "hidden": false
    },
    {
      "key": "PROGRESS",
      "label": "Progress",
      "type": "string",
      "description": "Short badge with the number of images in the latest batch and the number sent across all runs.",
      "artifact": false,
      "badge": true,
      "badgeOpens": null,
      "hidden": false
    },
    {
      "key": "BATCH_COUNT",
      "label": "Images in batch",
      "type": "string",
      "description": "Number of images in the latest batch, as a plain number to compare in a Condition. 0 when no unsent image was left.",
      "artifact": false,
      "badge": false,
      "badgeOpens": null,
      "hidden": false
    },
    {
      "key": "SENT_ASSETS",
      "label": "Sent Assets",
      "type": "dataset",
      "description": "ID of the dataset containing every image this node has sent across runs.",
      "artifact": false,
      "badge": false,
      "badgeOpens": null,
      "hidden": true
    }
  ]
}