Custom Components in Gradio

[abidlabs]

Converted to a tracking + discussion issue for custom components.

Context

to follow

Parts

• FE support for arbitrary components being loaded + mounted

  • A way to register components of any description internally (no user impact)
  • A consistent (versioned) interface for components + access to app level metadata (minor)

• BE support for custom components

  • A way to generate a component for use in gradio (minor)
  • A consistent interface for all components (args, classes to extend, methods that must be present, urils etc that can be used by authors (di or import))

• A way to share custom components

  • See theme sharing proposal
  • More complex than themes as the JS etc cannot be generated by the BE and passed to the frontend
  • Svelte or any? Build process? action/ build server?

• A way to consume custom components

  • APIs
  • CDN or self host?

• DX

  • how will a user develop and test their custom component?
  • How will a user use theme values such that their custom component will automatically be themed when a user passes in a new theme?
  • Can be make utils/ components/ whatever available to users as building blocks for their components?
  • How do we document all of this stuff?

I envisage many kinds of custom components:

• fully custom with a new python + html/js/css
  • either extending from existing gradio components or fully custom implementations
• pure frontend components that use some gradio python class. Basically a new frontend for existing components
• Pure python components.
  • Something like the above, composition for components with a new interface
  • or new python implementation for an existing component that will use an existing frontend. (Maybe is changes stuff like preprocessing or w/e)

These are the core usecases I think we need to cover.

More detail to follow

Original

We've heard requests from many different places (Discord, #1410) on folks who want to create plugins / custom components in Gradio. We should think about how we want to support this (do we want plugins that are not part of the main library?) and write up a Guide showing how to contribute these kinds of pulgins / components.

cc @pngwn

[omerXfaruq]

Custom component guides seems straightforward to me, but not sure about the plugin part. Could we provide a design that would allow users to create plugins with any use-case, guess not? Wouldn't it be more meaningful for users to add components when they need a specific use-case instead of creating plugins?

Though users being able to interact with components like in #1410 would be very cool, though have no idea about how to support it. Leaving it to our frontend masters, @dawoodkhan82, @aliabid94, @pngwn .

[jrabary]

I'm trying to understand the structure of Gradio in order to build a quick proof of concept of what we want to achieve in the issue #1752. I'm wondering why there are two definitions of each components. One under packages and one a wrapper defined in app/components ?

[jrabary]

Some ideas that I'm testing on my side in order to "plugin" a custom component developed outside Gradio repository

In the python side, define the component as below

from gradio.events import Changeable, Submittable
from gradio.components import IOComponent
from typing import Any, Optional


class CustomComp(Changeable, Submittable, IOComponent):
    def __init__(
        self,
        value: str = None,
        *,
        label: Optional[str] = None,
        show_label: bool = True,
        interactive: Optional[bool] = None,
        visible: bool = True,
        elem_id: Optional[str] = None,
        **kwargs,
    ):

    ....

Then use it as usual

def update(name):
    return f"Welcome to Gradio, {name}!"


with gr.Blocks() as demo:
    gr.Markdown("Start typing below and then click **Run** to see the output.")
    with gr.Row():
        inp = gr.Textbox(placeholder="What is your name?")
        out = gr.Textbox()

    with gr.Row():
        custom = CustomComp()

    btn = gr.Button("Run")
    btn.click(fn=update, inputs=inp, outputs=out)

demo.launch()

The backend seems to be OK with that.

Of course the frontend is missing the definition of customcomp. At this point, we need to update the component_map of the Gradio App but without touching the Gradio Repo. Two things, that I see, are missing in order to do that:

• a way to load to the frontend the .js file containing the CustomComp definition. Maybe as an optional argument of the python application.
• a way to dynamically update the component_map if the above optional argument is enabled

What is your opinion about this approach @abidlabs @freddyaboulton @farukozderim ?

[pngwn]

@jrabary There aren't two definitions, we have just split out the core component from the 'gradio' component that comes with a lot of app specific concerns. This is so we can publish the core components in the future and they have a sensible, generic API. It is just an implementation detail and doesn't have much impact on this feature.

What is your opinion about this approach?

I don't think it needs to be this verbose for users in the simple case. All we need is a way to register the component somehow, they are relatively decoupled from everything else. Can't we just define a class factory so users can do something like:

custom = CreateCustomComponent(
  name="whatever", 
  location="whatever", 
  value="whatever",
  prop1="whatever", 
  prop2="whatever"
)

This could even accept a class to extend from without needing to actually define a whole new class. Maybe they just want custom version of a certain component, for example (this feels like the most common case).

From this we can treat it as a component, pre and post processors would just return self.value, other methods would need some sensible default that is basically just an identity function with bells and whistles (serisalisation/deserialisation, etc). For more advanced uses cases we can allow users to extend any Class they want but we still need some kind of special class/ function so that we can distinguish the 'custom' component. We would need to give them a special key of some description in the config in order to handle the component mapping + loading.

Something like this might do for more complex use cases:

class Custom(CustomComponent):
 ...

Or if a use wants to inherit some of the behaviour of other classes:

class Custom(CustomComponent, Radio):
 ...

I'm not really sure that we want to expose every internal class for users to extend, this will increase our API surface area pretty significantly and make breaking changes far more likely. We should maintain a seperate class or set of classes for this purpose to act as an abstraction layer between the internal + external APIs. Even if they are the same now it will give us more flexibility and freedom in the future.

Maybe the CustomComponent extension isn't strictly necessary but it does make for a very explicit API.

It also isn't clear to me where this component would load from (do users self host or do we host it in the gradio app?), how we design the API (when we ship this what is now an internal API becomes public, we'll need to go over the current API carefully), how we guide users to create components (they are built with svelte which requires a bunch of tooling to compile and must be compiled in a specific way to work with gradio).

I don't have the bandwidth to look into these issues right now but will make some time as soon as I can.

[jrabary]

We may borrows some ideas from dash to handle this custom components issue. They describe here https://github.com/plotly/dash-component-boilerplate and here https://dash.plotly.com/plugins how to write custom components. https://github.com/plotly/dash-deck is an example of components developed by third party in a separate repo.
Here is what I learn by reading their repo (not sure, I got it right):

• components are developed only in react their frontend framework
• the python library extension add all of its frontend assets path to the main python server so it can be server along side all the core assets
• assets come with the extension library
• dash python server has the list of all available components - frontend assets that can be loaded in the frontend

[omerXfaruq]

@jrabary I think we should also take a look at what we need for Custom Components.

1. Are we just trying to change the Backend functionalities(this would be easier)
2. Are we trying to create components with unique frontend designs or functionalities(this would require a development from scratch at backend and frontend.

For supporting 1, we could design or use a very generalistic component which can support a lot of use-cases, and make it extendible or usable in Backend, maybe?

How does this sound @jrabary?

[jrabary]

@farukozderim For me it's more the second option. Here is an example to explain what I have in mind. Currently there is a Component in Gradio that display image and eventually has a crop functionally. What if I want to build a Gradio App that show up an inpainting algorithm ? I would like to display an image and erase some part of it with a brush or to draw some noise with a pen. I'm not sure this feature is supported currently by the Image component and the known way to add this feature is to update the component inside the Gradio codebase. I would like to build such component in its own repo and maybe provide a full set of components library that we can plug inside a Gradio app.

Some how the idea can be applied to the Gradio core component as well. Like, we want to separate them into a different group and import only the group we need:

from gradio import core_components # will import component like Block, ....
from gradio import image_components # will import component like Image

from custom_components import my_component # will import a custom component

[omerXfaruq]

OK! Then I think you can follow this guide, I think it would solve your need. Could you also drop feedback about the guide in a new issue, how clear was it, and were you able to easily create a new component? Or was there anything missing?

[omerXfaruq]

Since we have a custom component guide, converted the title to just Plugins.
If a need arise in the future, we would just move the custom components into a different file.

[jrabary]

OK! Then I think you can follow this guide, I think it would solve your need. Could you also drop feedback about the guide in a new issue, how clear was it, and were you able to easily create a new component? Or was there anything missing ?

Thanks for the guide. The guide is pretty clear and answers some of my questions. The thing is, I would like to build my components inside its own repo. Not in the Gradio codebase. My understanding is that I need to add the new components inside gradio source directly and edit internal gradio files like components.py or directory.ts. If I want it to be available publicly I will need to ask you to merge it into the Gradio main branch.

Being able to do the same thing but outside the Gradio source code would be great. Just plug the new components by importing python library and javascript files corresponding to the components.

I see two things that are needed to achieve this:

• being able to send from the backend metadata about the plugin to use (name, js asset path)-mapping at least
• being able to load dynamically the additional plugin at the frontend to make their definition available. This one might be tricky. I don't know if we can do it easily with svelte. Maybe add a Svelte component that load dynamically other custom element based on the metadata

[abidlabs]

Hi @jrabary, just to chime in here: On one hand, we actually do have support for inpainting demos, using the Image component with the tool set to sketch. So:

𝚐𝚛.𝙸𝚖𝚊𝚐𝚎(𝚝𝚘𝚘𝚕="𝚜𝚔𝚎𝚝𝚌𝚑")

Here's an example demo: https://huggingface.co/spaces/akhaliq/lama
With code: https://huggingface.co/spaces/akhaliq/lama/blob/main/app.py#L35

But to your larger point about supporting custom components outside of the library, we fully agree that this is something we need to support. It's a fair bit of work, but it is definitely something on our roadmap! (You can follow this issue for updates)

[space-nuko]

Just for the sake of updates, I was wondering if there has been any progress on this feature? There have been a lot of recent developments where the lack of custom components makes some workflows more inconvenient:

• Model browser
• Controlling OpenPose skeletons
• Image segmentation being able to paint separate parts of an image with different colors
• And of course all the other issues about custom components that have been opened already.

I could maybe help if there's anything actionable at this stage. But that's okay if things are still too early to implement yet.

And thanks to the gradio team for all your continuing hard work so far, without you all the current revolution in generative tech could not have happened

[pngwn]

@space-nuko That's great to hear. This hasn't been a priority up to this point as we worked through theming (which will be with us shortly). But we are actively discussing this at the minute. As soon as we have mapped out this feature, we'll add as much detail as we can to an issue (either this one or a new one) and start to get some feedback from the community to help shape this feature.

It's a pretty large feature with lots of moving parts but we'll be starting on it soon and are keen to get as much feedback as possible from the community, as well as contributions where that makes sense!

[abidlabs]

Let's close this issue and use #5564 to track any issues related to custom components