RFC: oEmbed support

[MatMaul]

https://oembed.com/
I was thinking about implementing oEmbed support.

I don't really know if I should just fully implement that in the client or prefetch embedding data in Matrix server.

URL previews use a Matrix service to fetch the preview, is it for client privacy reasons ?
Does it really apply to embedding use cases since a call to the provider will be made anyway to fetch the embedded resource ?

If we involve the server, we can prefetch oembed url or the full oembed properties.

From my point of view I think we should:

• prefetch the oembed url in the url preview service and add it in a property of the answer (og:oembed for example), I think it makes sense since the preview service needs to fetch and parse the original url anyway
• fetch the oembed url from the client and get final image URL or html code to embed the video player, and inject it in the discussion thread.

We can also call the oembed url directly in the url preview service and put oembed properties in the preview response.
I don't think it is a great idea however, we don't gain much client privacy since we will embed the resource anyway, and we may want to adjust some maxheight/maxwidth parameter that depend on the client.

[t3chguy]

URL previews use a Matrix service to fetch the preview, is it for client privacy reasons ?

It prevents leaking the Client IP to the potentially untrusted service

[MatMaul]

Ok that was my guess.

Last random thought: we could also get the resource to embed in the server and pass it to the client, however it is only feasible with photos which have direct URL, oEmbed for videos only gives you some html code to inject. And to be fair videos is the main use case, not photos :)

[MatMaul]

Up !

I have 2 working prototypes:

• one with html oembed code fetched from the server using oembed directly. I found it a bit dangerous since we are injecting unstructured and unchecked html/js code coming from the server.
• one fully client side using noembed.com. Oembed can't be used directly client side because most of the providers (youtube included...) don't include any lax CORS policy on their oembed endpoint...

[lampholder]

Hey @MatMaul - thinking about the client privacy issue, does your prototype involve hitting noembed.com/the oembed-compatible service from the client directly, or have you been making modifications to the URL-preview stuff?

[MatMaul]

I have prototypes for both, one which fetches the embedding code server side and one completely client side.
If we consider the Matrix server fully trusted we can do what Slack seems to be doing currently: get the embedding data server side, display the image preview, and we actually load the html only when clicking the preview. With that mechanism the foreign html is only injected after a user action, and we can also add a warning per domain for example, something like "This will load content from youtube.com, do you trust this website ?" with a "Always trust for this website" option or something like that.

[clokep]

I think this is essentially going to be implemented by matrix-org/synapse#7920.

[brendanheywood]

I'm very interested in this. Note we want to use it for internal systems and its important that its done client side using an iframe embed inside the client for a few reasons:

1. we want the embed to always be current state
2. each persons view of the embed might be different, ie some people may not even be able to see inside the resource

I totally get the security angle, but the way we've mitigate that before is there is an explicit admin setting which whitelists which oembed urls can be shown.

@MatMaul can you share a PR for both variants?

[turt2live]

This sort of change is up to the server and not us. Synapse has some oEmbed support these days, and matrix-media-repo fully supports it.