Support controllable RGB points or brushes for Image inout.

[HighCWu]

• I have searched to see if a similar issue already exists.

Is your feature request related to a problem? Please describe.

I'm always frustrated when I try to make a inference to input and output points coordinates or RGB color blocks.

I want to know if it is possible to add controllable points to an input images. Like detect face 68 feature points and display the points. When meet error point positions, user can move the points. Or something like rgb hint points for image colorization.

More broadly, probably like labelled brushes in GauGAN.

Describe the solution you'd like

• Controllable points component.
  It would be like:

input_image = gradio.Image()

# interactive means movable
points = gradio.Points(component=input_image, interactive=True, numbers=68) 

output_image = gradio.Image()
btn1 = gradio.Button()
btn2 = gradio.Button()
btn1.click(detect_face_68points, inputs=[input_image], outputs=[points])
btn2.click(model_inference, inputs=[input_image, points], [output_image])

Application scenarios: Fix the output of the first stage detection model.

I want to make a demo for my HighCWu/face_morpher_plus

• Controllable colorful points component for colorize task.
  It would be like the UI in junyanz/interactive-deep-colorization

• Canvas brushes for different colors of different label
  It would be like the UI in NVlabs/GauGAN

• The above suggestions seem to make Gradio heavy. So I suggest exposing some interface for users to write plugins.

It would be something like:

html = gradio.HTML('''
<mycanvas id="my-canvas">...</mycanvas>
<script>
mycanvas.setInput(gr.GetComponentById('input_image'))
mycanvas.setPoints(gr.GetComponentById('points'))
mycanvas.setTrigger(gr.GetComponentById('btn_1')) // trigger click event when upload finished
</script>
''')
input_image = gradio.Image(id='input_image', visible=False)
points = gradio.Variable(id='points')
output_image = gradio.Image()
btn1 = gradio.Button(id='btn_1', visible=False)
btn2 = gradio.Button()
btn1.click(detect_face_68points, inputs=[input_image], outputs=[points])
btn2.click(model_inference, inputs=[input_image, points], [output_image])

[omerXfaruq]

Thx for the suggestion @HighCWu!

These look like a beautiful suggestions, we might think over how to support these use-cases. @aliabid94, @pngwn, @dawoodkhan82, @abidlabs.

Btw I know that we support custom javascript calls in the library, but not sure if it supports this use-case, probably not.

[HighCWu]

Btw I know that we support custom javascript calls in the library, but not sure if it supports this use-case, probably not.

@farukozderim I looked at the code you mentioned. It's more like calling a frontend function before the event is triggered, rather than the frontend triggering the event. As for how the user writes the frontend code to get the gradio components on
html js code, I didn't find it from the documentation.

[abidlabs]

Hi! We've now made it possible for Gradio users to create their own custom components -- meaning that you can write some Python and JavaScript (Svelte), and publish it as a Gradio component. You can use it in your own Gradio apps, or share it so that anyone can use it in their Gradio apps. Here are some examples of custom Gradio components:

• A "Rich Textbox" that allows you to write bold/italics/colored text
• A "Folium Map Viewer" component that allows you to use interactive maps

You can see the source code for those components by clicking the "Files" icon and then clicking "src". The complete source code for the backend and frontend is visible. In particular, its very fast if you want to build off an existing component. We've put together a Guide: https://www.gradio.app/guides/five-minute-guide, and we're happy to help if you'd like to put together a custom component for this issue

[HighCWu]

I'm very happy that Gradio now supports users to customize components, but I also want to know whether Gradio's native drawing board supports the original image and new strokes being transmitted to the backend separately. For example, I drew colored strokes on a grayscale image, and I want to get both the original image and the strokes drawn by the user in the backend. I think this function should be expanded on the existing drawing board of Gradio.

[abidlabs]

This will be possible in the new ImageEditor we are releasing as part of Gradio 4.0 -- you can follow along here: #6169

[HighCWu]

This will be possible in the new ImageEditor we are releasing as part of Gradio 4.0 -- you can follow along here: #6169

Great job. I will follow up

[abidlabs]

I'll go ahead and close this issue since we are not planning to include this in the core Gradio library. But happy to help if you are interested in making this a custom Gradio component (feel free to ask questions in this issue).