Skip to navigation
Node LibraryImport

Import from cloud storage

Overview

Imports images or videos from a folder of an S3-compatible bucket into the OvalBee dataset you pick. Annotations stored next to the media come along, and their format is recognized from the folder contents.

Works with AWS S3 and with S3-compatible storage such as MinIO, Ceph or RustFS. The media are copied into OvalBee storage, so the dataset keeps working when the bucket changes.

Import from cloud storage Dataset

Prerequisites

RequirementWhat you need
S3 secretA secret of type S3 with an access key ID and secret access key that can list and read the bucket, plus its region or endpoint.
Network accessA worker that can reach the bucket’s endpoint.

How it works

1

Lists every file under the bucket folder without downloading anything.

2

Recognizes the annotation format from the file layout and the annotation files, or uses the format picked in Annotation format.

3

Downloads the media and annotation files that belong to the import and copies the media into OvalBee storage.

4

Adds the media and their annotations to the destination dataset and adds the classes and tags the annotations declare to that dataset’s ontology.

FAQ

For images: Supervisely, COCO, YOLO, Pascal VOC, LabelMe and Cityscapes. For videos: Supervisely. A folder with media and no annotation files in a known format is imported as plain images or videos.

Pick the format in Annotation format instead of Auto. The node then reads the folder as that format and skips detection.

Turn off With annotations. Every image or video under the folder is imported, and annotation files are ignored.

For AWS, the region is enough and the endpoint can stay empty. For MinIO and other S3-compatible storage, set the endpoint and turn on Force path style. The endpoint never includes the bucket name: use https://s3.eu-central-1.amazonaws.com, not https://my-bucket.s3.eu-central-1.amazonaws.com. IAM role secrets aren’t supported yet.

The secret may not have permission to list or read the bucket, the bucket path may point at an empty folder, or no image or video was found under it. For videos, only Auto and Supervisely are accepted in Annotation format.

The folder is imported again, and every file gets a second copy in the destination dataset. Point the node at a new dataset when you don’t want duplicates.

Inputs

S3 credentials
secretRequired

S3 secret with an access key pair and a region or endpoint. Works with AWS S3 and S3-compatible storage such as MinIO. Allowed credential types: s3. Key: S3_SECRET.

Bucket path
stringRequired

Bucket and folder to import from, for example s3://my-bucket/datasets/cars/. Every file under the folder is scanned. Placeholder: s3://my-bucket/path/to/dataset/. Key: PATH.

Media type
selectDefaults to imagesRequired

Whether the folder holds images or videos. Key: MEDIA_TYPE.

Options:

  • Images (images)
  • Videos (videos)
With annotations
booleanDefaults to trueRequired

Import annotations next to the media. Turn off to import only the media files. Key: WITH_ANNOTATIONS.

Annotation format
selectDefaults to autoRequired

Format of the annotations. Auto detects it from the folder contents. Videos support Auto and Supervisely only. Visible when WITH_ANNOTATIONS is not False. Key: ANNOTATION_FORMAT.

Options:

  • Auto (auto)
  • Supervisely (supervisely)
  • COCO (coco)
  • YOLO (yolo)
  • Pascal VOC (pascal_voc)
Destination dataset
datasetRequired

Dataset that receives the imported media and annotations. Type a new name to create it. Key: DATASET.

Outputs

Dataset
dataset

ID of the OvalBee dataset that received the imported media. Key: DATASET.

JSON config

Machine-readable node interface for automation and advanced usage.

{
  "name": "Import from cloud storage",
  "description": "Import images or videos with their annotations from an S3-compatible bucket. The annotation format is detected automatically.",
  "category": "Import",
  "namespace": "ovalbee",
  "templateKey": "import_ops/import_from_cloud_storage",
  "version": "v1",
  "inputs": [
    {
      "key": "S3_SECRET",
      "label": "S3 credentials",
      "type": "secret",
      "description": "S3 secret with an access key pair and a region or endpoint. Works with AWS S3 and S3-compatible storage such as MinIO.",
      "required": true,
      "default": null,
      "visibleWhen": null,
      "options": {
        "types": [
          "s3"
        ]
      }
    },
    {
      "key": "PATH",
      "label": "Bucket path",
      "type": "string",
      "description": "Bucket and folder to import from, for example `s3://my-bucket/datasets/cars/`. Every file under the folder is scanned.",
      "required": true,
      "default": null,
      "visibleWhen": null,
      "options": {
        "placeholder": "s3://my-bucket/path/to/dataset/"
      }
    },
    {
      "key": "MEDIA_TYPE",
      "label": "Media type",
      "type": "select",
      "description": "Whether the folder holds images or videos.",
      "required": true,
      "default": "images",
      "visibleWhen": null,
      "options": {
        "control": "tabs",
        "options": [
          {
            "label": "Images",
            "value": "images"
          },
          {
            "label": "Videos",
            "value": "videos"
          }
        ]
      }
    },
    {
      "key": "WITH_ANNOTATIONS",
      "label": "With annotations",
      "type": "boolean",
      "description": "Import annotations next to the media. Turn off to import only the media files.",
      "required": true,
      "default": true,
      "visibleWhen": null
    },
    {
      "key": "ANNOTATION_FORMAT",
      "label": "Annotation format",
      "type": "select",
      "description": "Format of the annotations. Auto detects it from the folder contents. Videos support Auto and Supervisely only.",
      "required": true,
      "default": "auto",
      "visibleWhen": {
        "key": "WITH_ANNOTATIONS",
        "operator": "notEquals",
        "value": false
      },
      "options": {
        "options": [
          {
            "label": "Auto",
            "value": "auto"
          },
          {
            "label": "Supervisely",
            "value": "supervisely"
          },
          {
            "label": "COCO",
            "value": "coco"
          },
          {
            "label": "YOLO",
            "value": "yolo"
          },
          {
            "label": "Pascal VOC",
            "value": "pascal_voc"
          }
        ]
      }
    },
    {
      "key": "DATASET",
      "label": "Destination dataset",
      "type": "dataset",
      "description": "Dataset that receives the imported media and annotations. Type a new name to create it.",
      "required": true,
      "default": null,
      "visibleWhen": null
    }
  ],
  "outputs": [
    {
      "key": "DATASET",
      "label": "Dataset",
      "type": "dataset",
      "description": "ID of the OvalBee dataset that received the imported media.",
      "artifact": false,
      "badge": false,
      "badgeOpens": null,
      "hidden": false
    }
  ]
}