ConceptsData

Datasets and versions

Named, versioned sets of images, videos and their annotations.

A dataset is a named set of assets: images or videos together with their annotations. Almost every node reads a dataset, writes one, or both. Labeling queues, train and validation splits, predictions, accepted and rejected work — they’re all datasets.

A dataset doesn’t copy files. It lists assets that live in the workspace’s storage, so the same image can be a member of the source dataset, a labeling queue and the final training set at once without taking up space three times.

What’s in a dataset

  • Media — images or videos.
  • Annotations — one annotation per image or video, holding every object and tag on it. Annotations travel with their images: when a node samples, splits, filters or moves images, their annotations come along.
  • An ontology — the class and tag vocabulary the annotations are written in. See Ontologies.
  • Tags on its assets, such as the split an image came from, for filtering.

Versions

Datasets have versions. Version 1 is created with the dataset, and later versions are added on top: v2, v3 and so on. Nothing is ever merged into an older version or rolled back in place — the history only grows.

Most versions are added by nodes:

  • A node’s own result. When a node creates a dataset for its output, the first run creates it and every rerun adds a new version to it. The dataset keeps the node’s name, and older results stay in its history instead of being overwritten.
  • Imports. Each drop into Import drag & drop and each run of the Supervisely Import images node adds a new version of the destination dataset.

Nodes that reorganize data you pointed them at — moving images between datasets, removing assets, copying into a destination — change that dataset in place. The node pages in the Node Library say which kind each node is.

Latest or pinned

A dataset input in a node can refer to a dataset in two ways:

  • Latest. The row marked @latest in the dataset picker follows the newest version. Each time the node runs, it uses whatever the newest version is at that moment.
  • Pinned. Expand a dataset that has several versions and pick one. The node keeps using that exact version, however many are added later.

Use latest for work that should keep up with the data: training on a growing labeled set, labeling whatever arrives next, reporting on the current state. Pin a version when the point is that it doesn’t change: a benchmark set that several models are compared on, a frozen validation split, a dataset a result was published from.

Creating datasets

You rarely create a dataset on its own. In any dataset picker, type a new name and press Create — the dataset is created on the spot and the input refers to it. The Dataset node does the same as a standalone step, and is a good way to give a flow one place where its main dataset is chosen.

Finding datasets

Storage in the main sidebar lists the workspace’s datasets with their newest version, ontology, number of assets, size and tags. Click a dataset to see its assets.

Some datasets are hidden from this list because they’re working state rather than results: the private queue of each labeler, the last-run snapshot some nodes keep next to their result, pools that nodes use to remember what they’ve already processed. They stay reachable from the nodes that use them.

In a flow, the dataset picker lists the datasets already used in the current flow first, then everything else under From Other Flows.