Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(Instance) segmentation annotation for images #1597

Open
1 of 4 tasks
dott1718 opened this issue Aug 19, 2022 · 0 comments
Open
1 of 4 tasks

(Instance) segmentation annotation for images #1597

dott1718 opened this issue Aug 19, 2022 · 0 comments
Assignees
Labels
feature Feature request ui Related to UI

Comments

@dott1718
Copy link

dott1718 commented Aug 19, 2022

The ui.image_annotator shall be extended by another feature: (instance) segmentation. This is very similar to the existing image box annotation feature, but allows polygon types and per pixel object annotations instead of boxes.

  • In case of instance segmentation, the output would be a list of objects with run length encoded masks or polygons or binary masks (raw data points) and a class descriptor (tag) for each object.
  • In case of segmentation, the output would be a list of classes (tags) with run length encoded masks or polygons or binary masks (raw data points) for each class.

The difference between the two is that Instance segmentation is the task of detecting and delineating each distinct object of interest appearing in an image vs. (semantic) segmentation which treats multiple objects within a single category as one entity.

For feature implementation, we can think about a polygon based (quick) and a brush based (pixel perfect) or lasso/pen tool (Labelbox pen tool) drawing approach.
To combine this feature with the existing box annotation feature, we are suggesting to add a tools and a task argument where developers can choose between rectangle, rectangle_4_point(see below), polygon, brush, pen and bounding_box, segmentation, instance_segmentation respectively.

Requirements:
Similar to the box component #1400, the segmentation annotation component should have three key properties:

  • image
  • tags: list of tags
  • items: list of polygons (polygon tool) or RLE masks (brush, pen) or binary masks (raw data points)

The image is displayed as a background. The tags are rendered as clickable elements that tag the highlighted areas. The items are rendered as semi-transparent colored regions and can be deleted.

Polygon tool: when selecting a single polygons it should be easy to

  • Drag and move the polygons
  • Move individual vertices
  • Add vertices
  • Delete the polygons
  • Change the tag
  • Zoom in and out of the image (preferably with mouse wheel). In the API, we would specify max width/height for the box and then the zoomed image would be rendered inside that box
  • Ability to respect prior polygons/masks (Labelbox - Draw to back)
    • Allow / disallow multiple annotations per pixel

Brush tool:

  • Additive or subtractive to current active class/instance
  • Ability to respect prior polygons/masks
  • Change the tag
  • Delete the mask

Pen tool:

  • Smart closing of paths (shall not need to be pixel perfect to work)
  • Ability to respect prior polygons/masks
  • Change the tag
  • Delete the mask

Examples of similar components:
image
image

@dott1718 dott1718 added the feature Feature request label Aug 19, 2022
@mturoci mturoci added the ui Related to UI label Sep 5, 2022
@mturoci mturoci self-assigned this Sep 9, 2022
mturoci added a commit that referenced this issue Sep 28, 2022
mturoci added a commit that referenced this issue Sep 28, 2022
mturoci added a commit that referenced this issue Sep 28, 2022
mturoci added a commit that referenced this issue Sep 28, 2022
mturoci added a commit that referenced this issue Sep 28, 2022
mturoci added a commit that referenced this issue Sep 28, 2022
mturoci added a commit that referenced this issue Sep 28, 2022
mturoci added a commit that referenced this issue Sep 28, 2022
mturoci added a commit that referenced this issue Sep 29, 2022
mturoci added a commit that referenced this issue Sep 29, 2022
mturoci added a commit that referenced this issue Sep 30, 2022
mturoci added a commit that referenced this issue Sep 30, 2022
mturoci added a commit that referenced this issue Sep 30, 2022
mturoci added a commit that referenced this issue Oct 3, 2022
mturoci added a commit that referenced this issue Oct 3, 2022
mturoci added a commit that referenced this issue Oct 3, 2022
mturoci added a commit that referenced this issue Oct 3, 2022
mturoci added a commit that referenced this issue Nov 3, 2022
mturoci added a commit that referenced this issue Nov 3, 2022
mturoci added a commit that referenced this issue Nov 3, 2022
mturoci added a commit that referenced this issue Nov 3, 2022
mturoci added a commit that referenced this issue Nov 3, 2022
mturoci added a commit that referenced this issue Nov 3, 2022
mturoci added a commit that referenced this issue Nov 3, 2022
mturoci added a commit that referenced this issue Nov 3, 2022
mturoci added a commit that referenced this issue Nov 3, 2022
mturoci added a commit that referenced this issue Nov 3, 2022
mturoci added a commit that referenced this issue Nov 3, 2022
mturoci added a commit that referenced this issue Nov 3, 2022
mturoci added a commit that referenced this issue Nov 3, 2022
mturoci added a commit that referenced this issue Nov 3, 2022
mturoci added a commit that referenced this issue Nov 3, 2022
mturoci added a commit that referenced this issue Nov 3, 2022
mturoci added a commit that referenced this issue Nov 3, 2022
mturoci added a commit that referenced this issue Nov 3, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature Feature request ui Related to UI
Projects
None yet
Development

No branches or pull requests

2 participants