(Instance) segmentation annotation for images

[dott1718]

The ui.image_annotator shall be extended by another feature: (instance) segmentation. This is very similar to the existing image box annotation feature, but allows polygon types and per pixel object annotations instead of boxes.

• In case of instance segmentation, the output would be a list of objects with run length encoded masks or polygons or binary masks (raw data points) and a class descriptor (tag) for each object.
• In case of segmentation, the output would be a list of classes (tags) with run length encoded masks or polygons or binary masks (raw data points) for each class.

The difference between the two is that Instance segmentation is the task of detecting and delineating each distinct object of interest appearing in an image vs. (semantic) segmentation which treats multiple objects within a single category as one entity.

For feature implementation, we can think about a polygon based (quick) and a brush based (pixel perfect) or lasso/pen tool (Labelbox pen tool) drawing approach.
To combine this feature with the existing box annotation feature, we are suggesting to add a tools and a task argument where developers can choose between rectangle, rectangle_4_point(see below), polygon, brush, pen and bounding_box, segmentation, instance_segmentation respectively.

Requirements:
Similar to the box component #1400, the segmentation annotation component should have three key properties:

• image
• tags: list of tags
• items: list of polygons (polygon tool) or RLE masks (brush, pen) or binary masks (raw data points)

The image is displayed as a background. The tags are rendered as clickable elements that tag the highlighted areas. The items are rendered as semi-transparent colored regions and can be deleted.

Polygon tool: when selecting a single polygons it should be easy to

• Drag and move the polygons
• Move individual vertices
• Add vertices
• Delete the polygons
• Change the tag
• Zoom in and out of the image (preferably with mouse wheel). In the API, we would specify max width/height for the box and then the zoomed image would be rendered inside that box
• Ability to respect prior polygons/masks (Labelbox - Draw to back)
  • Allow / disallow multiple annotations per pixel

Brush tool:

• Additive or subtractive to current active class/instance
• Ability to respect prior polygons/masks
• Change the tag
• Delete the mask

Pen tool:

• Smart closing of paths (shall not need to be pixel perfect to work)
• Ability to respect prior polygons/masks
• Change the tag
• Delete the mask

Examples of similar components:

• Polygon support feat: Polygon support for image annotator #1643
• Pen support
• Brush support
• RLE support