feat: Image annotator

[mturoci]

Demo

Screen.Recording.2022-05-04.at.3.05.02.PM.mov

Proposed API

/** Rectangle (box) annotation. */
interface ImageAnnotatorRect {
  /** Start X dimension. */
  x1: U
  /** End X dimension. */
  x2: U
  /** Start Y dimension. */
  y1: U
  /** End Y dimension. */
  y2: U
}

/** Defines a particular shape to be used for the annotation. */
interface ImageAnnotatorShape {
  rect: ImageAnnotatorRect
}

/** Image annotator tag. */
interface ImageAnnotatorTag {
  /** An identifying name for this tag. */
  name: Id
  /** Text to be displayed. */
  label: S
  /** HEX or RGB color string used as background for highlighted phrases. */
  color: S
}

/** Create an annotator item with initial selected tags or no tag for plaintext. */
interface ImageAnnotatorItem {
  /** The annotation shape. */
  shape: ImageAnnotatorShape
  /** Tag connected to the highlighted section. */
  tag: S
}

/**
 * Create an image annotator component.
 * 
 * The image annotator component enables user to manually annotate parts of an image.
 */
export interface ImageAnnotator {
  /** An identifying name for this component. */
  name: Id
  /** The image to annotate. */
  image: S
  /** The text annotator's title. */
  title: S
  /** List of tags the user can annotate with. */
  tags: ImageAnnotatorTag[]
  /** Existing annotations if any. */
  items?: ImageAnnotatorItem[]
  /** True if the form should be submitted when the annotator value changes. */
  trigger?: B
  /** The card’s image height. Intrinsic image size is used by default. */
  image_height?: S
}

Returning ImageAnnotatorItem .

@lo5 please have a look at the API - I tried to design it in a way so that we can support multiple shapes (circles/paths) in future as well. Let me know if you see room for improvement though.

Notes

• Unit tests / docs to be delivered once the API is settled.

cc @dott1718 for shallow feedback based on just video. I expect more detailed feedback after you play around with it in nightly.
Closes #1323

[dott1718]

Thank you @mturoci, the component looks great! We will provide more feedback as soon as we can try it out.

[lo5]

Demo looks great 🎸!

It's not clear to me how items and ImageAnnotatorItem, ImageAnnotatorShape, ImageAnnotatorRect are intended to be used. The one example in this PR doesn't use any of them. Maybe that'll help clarify why tag is a string, and why rect is an attribute of ~Shape. It would be good to add another example that show those components in use.

[mturoci]

It's not clear to me how items and ImageAnnotatorItem, ImageAnnotatorShape, ImageAnnotatorRect are intended to be used.

I updated the existing example to show predefined annotation. The basic reasoning is:

• Tag is a string so that we are consistent with tag in ui.text_annotator + there are not many other options.
• The ImageAnnotatorShape is a bit more tricky, but the overarching goal is to do us a favor and allow for extending the image annotator with other shapes in the future, e.g. circle/paths. I took this approach to be consistent with how we define our table cell types.

[mturoci]

@lo5 one more thing that is worth discussing is the image aspect ratio. Since images to be annotated can be various sizes, we have basically 2 options:

• Display them in their original (intrinsic) size even at the price of scrollbars (in case of overflow).
• Provide height attr and scale the image accordingly - I went for this option.

Since I chose the second option as it is more UX-friendly, I had to make adjustments to the coordinates based on the aspect ratio so that developers can specify and receive coordinates relative to the original size of their img. Example:

Original img: 100x100 with coordinates 10,10.
Scaled image with height 50px: 50x50 with coordinates translated to 5,5.

The proposed approach makes sense to me, but feel free to point out any alternatives as you see fit.

[lo5]

OK, so if I'm understanding this correctly, ImageAnnotatorItem.tag looks up an existing ImageAnnotatorTag.name to determine the label and color?

[lo5]

The proposed approach makes sense to me, but feel free to point out any alternatives as you see fit.

Your approach sounds good to me.

[mturoci]

OK, so if I'm understanding this correctly, ImageAnnotatorItem.tag looks up an existing ImageAnnotatorTag.name to determine the label and color?

That's right.

[mturoci]

Thanks for the thorough review @lo5! All the comments were addressed and the branch rebased. Could you please give it one last quick glance and if it looks good to you, we can merge.