Node LibraryEvaluate

Active learning curve

Overview

Records how mAP on a fixed labeled set grows as an active learning loop adds labeled images, one point per cycle. When a full-data evaluation is connected, the chart also shows it as the ceiling to aim for.

Place it once inside a looping flow instead of drawing every cycle by hand. Each run extends the same curve, and the Latest mAP output lets a Condition node keep the loop running until the model is good enough.

Train Evaluate Active learning curve Mine and label images

Prerequisites

RequirementWhat you need
Test evaluationAn object detection evaluation from the Evaluate object detection node, on the same labeled set every cycle.
Baseline evaluation (optional)An evaluation of a model trained on all available data, on that same set. Needed only for the reference line.

How it works

1

Reads the points recorded by earlier runs of this node.

2

Counts the images in the labeled training set and adds a point with this cycle’s mAP at that count.

3

Publishes a report with the curve, the baseline line and a per-cycle table, and outputs the latest labeled image count.

FAQ

The point for that evaluation is replaced, not duplicated. Only a new evaluation adds a point.

Wire Latest mAP into a Condition node, for example “Less than 0.9”. The loop continues while the latest mAP is below the target and stops once a cycle reaches it. If the curve flattens below the target, the loop keeps going until nothing is left to pick, so set a target the curve can reach.

Remove the node and add it again. The history lives in the node, so a fresh node starts from zero points.

The x axis lists the labeled image counts as labels, not on a numeric scale. Read the counts from the labels and the per-cycle table.

Inputs

Cycle evaluation
model_evaluationRequired

Evaluation of this cycle’s model on a fixed evaluation set, the same one every cycle. Each run adds it as one point, and running again on the same evaluation replaces that point instead of adding another. Key: EVALUATION.

Labeled training set
datasetRequired

Dataset this cycle’s model was trained on. Its image count is the point’s position on the x axis. Key: TRAIN_DATASET.

Full-data baseline
model_evaluation

Evaluation of a model trained on all available data, on the same evaluation set. Drawn as a dashed reference line the curve should approach. Key: BASELINE_EVALUATION.

Outputs

Learning curve report
report

Report with the curve and one row per cycle. A new report is published on every run. Shown as an artifact. Key: REPORT_ID.

Latest mAP
string

mAP 50-95 of the latest point, as a plain number such as 0.6496. Wire it into a Condition node to keep the loop running until the model reaches the quality you need. Shown on the node as a badge. Key: MAP.

Cycles
string

Number of points on the curve. Key: CYCLES.

History
object

Points recorded so far, kept so each run can extend the curve. Hidden from the flow editor. Key: HISTORY.

Models and configuration

There is nothing to configure. The mAP values come from the evaluations, so every cycle must be evaluated with the same settings on the same labeled set for the points to be comparable.

Report

Answers whether the next batch of labels is still worth it: a curve that flattens below the full-data line means each cycle buys less.

  • Summary: latest mAP, labeled images, number of cycles and, with a baseline, the gap to it.
  • mAP by labeled images: mAP 50-95 and mAP 50 per cycle, with the baseline as dashed lines.
  • Cycles: one row per cycle with its image count, both mAP values, the change against the cycle before, and the model version.

Runtime

Runs on an ordinary worker without a GPU. Counting the training set is the only work that grows with data size.

JSON config

Machine-readable node interface for automation and advanced usage.

{
  "name": "Active learning curve",
  "description": "Record one point per active learning cycle and chart mAP on a fixed evaluation set against the number of labeled training images.",
  "category": "Evaluate",
  "namespace": null,
  "templateKey": "evaluate/active_learning_curve",
  "version": "v1",
  "inputs": [
    {
      "key": "EVALUATION",
      "label": "Cycle evaluation",
      "type": "model_evaluation",
      "description": "Evaluation of this cycle's model on a fixed evaluation set, the same one every cycle. Each run adds it as one point, and running again on the same evaluation replaces that point instead of adding another.",
      "required": true,
      "default": null,
      "visibleWhen": null,
      "options": {
        "task_type": "object_detection"
      }
    },
    {
      "key": "TRAIN_DATASET",
      "label": "Labeled training set",
      "type": "dataset",
      "description": "Dataset this cycle's model was trained on. Its image count is the point's position on the x axis.",
      "required": true,
      "default": null,
      "visibleWhen": null,
      "options": {
        "creatable": false
      }
    },
    {
      "key": "BASELINE_EVALUATION",
      "label": "Full-data baseline",
      "type": "model_evaluation",
      "description": "Evaluation of a model trained on all available data, on the same evaluation set. Drawn as a dashed reference line the curve should approach.",
      "required": false,
      "default": null,
      "visibleWhen": null,
      "options": {
        "task_type": "object_detection"
      }
    }
  ],
  "outputs": [
    {
      "key": "REPORT_ID",
      "label": "Learning curve report",
      "type": "report",
      "description": "Report with the curve and one row per cycle. A new report is published on every run.",
      "artifact": true,
      "badge": false,
      "badgeOpens": null,
      "hidden": false
    },
    {
      "key": "MAP",
      "label": "Latest mAP",
      "type": "string",
      "description": "mAP 50-95 of the latest point, as a plain number such as 0.6496. Wire it into a Condition node to keep the loop running until the model reaches the quality you need.",
      "artifact": false,
      "badge": true,
      "badgeOpens": null,
      "hidden": false
    },
    {
      "key": "CYCLES",
      "label": "Cycles",
      "type": "string",
      "description": "Number of points on the curve.",
      "artifact": false,
      "badge": false,
      "badgeOpens": null,
      "hidden": false
    },
    {
      "key": "HISTORY",
      "label": "History",
      "type": "object",
      "description": "Points recorded so far, kept so each run can extend the curve.",
      "artifact": false,
      "badge": false,
      "badgeOpens": null,
      "hidden": true
    }
  ],
  "automation": {
    "requires_configuration": true
  },
  "widgets": {
    "widget": {
      "id": "report-preview",
      "settings": {
        "reportAsset": {
          "type": "variable",
          "value": "self.outputs.REPORT_ID"
        },
        "name": {
          "type": "input",
          "value": "Open learning curve"
        }
      }
    }
  }
}

References