ml5-extra-imagesegmentation

This is an experiment in incorporating transformers.js into ml5.js for use with p5.js. Done by @gohai, @chanel1130, @Lisa-HuangZijin and @Ricci-Liu in Summer of 2024.

We're making use of the detr-resnet-50-panoptic image segmentation model here, which we found to provide acceptable performance in the browser (albeit not quite fast enough to run it continuously from a webcam).

Examples

• Basic
• Webcam Collage by Chanel
• Webcam Drag & Drop Storytelling by Lisa
• Webcam Grid Storytelling by Lisa
• Webcam Parallax by Ruiqi

Usage

In the head section of your index.html, add this line after the line that loads ml5.js:

<script src="https://unpkg.com/[email protected]/src/ml5-extra-imagesegmentation.js"></script>

Load the image segmenter like so:

let segmentation;

function preload() {
  segmentation = ml5.imageSegmentation({ feature_extractor_size: 256 });
}

Omitting the feature_extractor_size option will improve the accuracy over speed.

To start a detection, pass an image or video to the detect() method, together with the name of a callback function.

segmentation.detect(img, gotResults);

Once the segmentation has finished, the callback function is called with an array of segments as argument.

function gotResults(results) {
  for (let i=0; i < results.length; i++) {
    result = results[i];
    console.log(result);
  }
}

Each result has the following properties: label, score as well as a mask image. The mask is ready to be used with p5's mask() method.

Acknowledgements

Many thanks to @xenova - first and foremost for transformers.js, but also for kindly providing feedback during the research for this project!