Skip to navigation
Node LibraryExport

Export to cloud storage

Overview

Exports a dataset’s images, and their annotations, to a folder in an S3-compatible bucket, laid out in the format you pick.

Works with AWS S3 and with S3-compatible storage such as MinIO, Ceph or RustFS. The export is the counterpart of Import from cloud storage: a folder written here can be read back by that node.

Dataset Export to cloud storage

Prerequisites

RequirementWhat you need
Dataset with imagesAn OvalBee dataset that already holds at least one image.
S3 secretA secret of type S3 with an access key ID and secret access key that can write to the bucket, plus its region or endpoint.
Network accessA worker that can reach the bucket’s endpoint.

How it works

1

Reads the source dataset and confirms it holds at least one image.

2

Opens a connection to the bucket folder from Bucket path, using the S3 secret.

3

Exports the images, and their annotations unless With annotations is off, laid out as Annotation format.

4

Applies If files exist to any file already sitting at the destination.

FAQ

YOLO writes images/train, labels/train and a data.yaml class list. COCO writes images/train and an annotations/train.json. Supervisely writes a project-level meta.json plus <dataset>/img and <dataset>/ann folders.

Turn off With annotations. The node still writes the layout for the picked Annotation format, but skips every annotation file.

Skip existing, the default, leaves an image alone once a same-sized file is already there and rewrites the annotation files, so a rerun only sends what changed. Overwrite replaces every file. Fail stops the run instead of writing into a folder that already holds anything.

For AWS, the region is enough and the endpoint can stay empty. For MinIO and other S3-compatible storage, set the endpoint and turn on Force path style. The endpoint never includes the bucket name: use https://s3.eu-central-1.amazonaws.com, not https://my-bucket.s3.eu-central-1.amazonaws.com. IAM role secrets aren’t supported yet.

The source dataset may have no images, the secret may not have permission to write to the bucket, the bucket path may be malformed, or If files exist may be set to Fail while the destination folder already holds files.

Inputs

Dataset
datasetRequired

Dataset whose images are exported. Key: DATASET.

S3 credentials
secretRequired

S3 secret with an access key pair and a region or endpoint. Works with AWS S3 and S3-compatible storage such as MinIO. Allowed credential types: s3. Key: S3_SECRET.

Bucket path
stringRequired

Bucket and folder to export to, for example s3://my-bucket/datasets/cars/. Placeholder: s3://my-bucket/path/to/export/. Key: PATH.

With annotations
booleanDefaults to trueRequired

Export annotations next to the images. Turn off to export only the image files. Key: WITH_ANNOTATIONS.

Annotation format
selectDefaults to superviselyRequired

Format the annotations are written in. Visible when WITH_ANNOTATIONS is not False. Key: ANNOTATION_FORMAT.

Options:

  • Supervisely (supervisely)
  • COCO (coco)
  • YOLO (yolo)
If files exist
selectDefaults to skipRequired

What to do when a file already exists at the destination. Key: IF_EXISTS.

Options:

  • Overwrite (overwrite)
  • Skip existing (skip)
  • Fail (fail)

JSON config

Machine-readable node interface for automation and advanced usage.

{
  "name": "Export to cloud storage",
  "description": "Export a dataset's images, with their annotations, to an S3-compatible bucket.",
  "category": "Export",
  "namespace": "ovalbee",
  "templateKey": "export_ops/export_to_cloud_storage",
  "version": "v1",
  "inputs": [
    {
      "key": "DATASET",
      "label": "Dataset",
      "type": "dataset",
      "description": "Dataset whose images are exported.",
      "required": true,
      "default": null,
      "visibleWhen": null
    },
    {
      "key": "S3_SECRET",
      "label": "S3 credentials",
      "type": "secret",
      "description": "S3 secret with an access key pair and a region or endpoint. Works with AWS S3 and S3-compatible storage such as MinIO.",
      "required": true,
      "default": null,
      "visibleWhen": null,
      "options": {
        "types": [
          "s3"
        ]
      }
    },
    {
      "key": "PATH",
      "label": "Bucket path",
      "type": "string",
      "description": "Bucket and folder to export to, for example `s3://my-bucket/datasets/cars/`.",
      "required": true,
      "default": null,
      "visibleWhen": null,
      "options": {
        "placeholder": "s3://my-bucket/path/to/export/"
      }
    },
    {
      "key": "WITH_ANNOTATIONS",
      "label": "With annotations",
      "type": "boolean",
      "description": "Export annotations next to the images. Turn off to export only the image files.",
      "required": true,
      "default": true,
      "visibleWhen": null
    },
    {
      "key": "ANNOTATION_FORMAT",
      "label": "Annotation format",
      "type": "select",
      "description": "Format the annotations are written in.",
      "required": true,
      "default": "supervisely",
      "visibleWhen": {
        "key": "WITH_ANNOTATIONS",
        "operator": "notEquals",
        "value": false
      },
      "options": {
        "options": [
          {
            "label": "Supervisely",
            "value": "supervisely"
          },
          {
            "label": "COCO",
            "value": "coco"
          },
          {
            "label": "YOLO",
            "value": "yolo"
          }
        ]
      }
    },
    {
      "key": "IF_EXISTS",
      "label": "If files exist",
      "type": "select",
      "description": "What to do when a file already exists at the destination.",
      "required": true,
      "default": "skip",
      "visibleWhen": null,
      "options": {
        "options": [
          {
            "label": "Overwrite",
            "value": "overwrite"
          },
          {
            "label": "Skip existing",
            "value": "skip"
          },
          {
            "label": "Fail",
            "value": "fail"
          }
        ]
      }
    }
  ],
  "outputs": []
}