Suggested Streams via Actor Definition

[evantahler]

Closes #10872, (inspired by https://github.com/airbytehq/oncall/issues/1303#issuecomment-1381772903 OC issue). Protocol Change Process Document.

What?

This PR allows connectors to suggest which streams are pre-selected for users creating a new connection. This is accomplished by adding a new “suggestedStreams" block to the source actor definitions. This change is backwards-compatible and does not require the re-publishing of existing connectors.

For many of our API sources, the connector provides a lot of streams. This is great because Airbyte connectors come with "batteries included" to enable the widest range of data-syncing use-cases. However, for some sources, most users probably don't want all of the streams. Some of those streams may be extraneous, but worse, they may be slow, expensive, and often make the syncing experience for the more "primary" streams worse. Perhaps this stream has a slower API, consumes more API "credits", or harms the global rate limit in a worse-than-normal way.

This PR allows each connector to tune the default stream selections to provide a better first-time sync experience.

The Convenience Principle: People will do what's convenient, and they will avoid what's inconvenient. All else being equal, to increase how many people do something, make it more convenient, and to decrease, make it less convenient

• @sherifnada

This PR comes with an example for soruce-github, a source with particularly egregious streams, which we don't think many people would opt-into.

Technical notes:

• No API or webapp changes needed
• All existing connectors will work the same way - defaulting to selecting all streams for a new connection
• All that connector developers need to do is supply a list of suggestedStreams in source_definitions.yaml if they want to select only some streams. Otherwise, all streams will remain suggested as they are today.

[evantahler]

Here's the new interface for connector developers who only want some streams selected by default

[malikdiarra]

I am a bit surprised that this is in the definition yaml file when the schema itself is not present in that file but requires calling a connector function and the suggestedStreams are definitely tied to the schema.

I'm not advocating moving that to a connector function but that might be a good signal that we would benefit from having the schema show up in the definition for API connectors.

[evantahler]

Yeah... that was the original proposal (#21437) complete with protocol change. For posterity, I added a comment in that PR as to why we chose to move this out of the protocol, and you can read more details here.

In the near future, all of "actor definitions" will be replaced with "connector metadata", and things will hopefully begin to make more sense.

[github-actions]

Airbyte Code Coverage

File	Coverage [84.5%]	🍏
V0_40_28_001__AddSuggestedStreams.java	100%	🍏
ConfigWriter.java	90.13%	🍏
DbConverter.java	79.6%	🍏

Total Project Coverage	24%	❌

[github-actions]

Platform Test Results

   234 files  ±0     234 suites  ±0   25m 16s ⏱️ +44s
1 612 tests ±0  1 601 ✔️ ±0  11 💤 ±0  0 ❌ ±0 
1 632 runs  ±0  1 621 ✔️ ±0  11 💤 ±0  0 ❌ ±0 

Results for commit 4ec6ba0. ± Comparison against base commit e39ee94.

♻️ This comment has been updated with latest results.

[timroes]

Just to double check (since I haven't ran this locally, just looked at the code): This selection only takes place when we newly create a connection for the first time, not every time we refresh the catalog? I.e. an already saved connection where the user does a "refresh from source" won't get all their stream selected states overwritten by this logic?

[evantahler]

Just to double check (since I haven't ran this locally, just looked at the code): This selection only takes place when we newly create a connection for the first time, not every time we refresh the catalog? I.e. an already saved connection where the user does a "refresh from source" won't get all their stream selected states overwritten by this logic?

Yep!
I've also added additional detail to the API response so in the future we can highlight "suggested by not selected" streams if we want

[erohmensing]

Just checking this out, looks exciting!

[erohmensing]

Thanks for thinking this through and laying it out, I'm sure it'll be helpful in some weird issue a year from now

[malikdiarra]

Can we add more metadata?
like a description or a title that would indicate what use case the suggested streams are trying to achieve?
It seems your github example exclude some streams related to workflow for example. Those can be useful to investigate CI.
(There might be some connectors that would also benefit from multiple suggestedStream with different use case each but that might be beyond the scope here)

[evantahler]

For sure! I added more context in c025313.

I like the idea of context-aware suggestions for the future! That's why this is an object. @michel-tricot and I brainstormed a streams-matcher array which might one day contain the logic for database sources to deselect tables with foreign keys, or any stream coming from a non-public namespace/scheams.

[malikdiarra]

Thanks for that, I think the idea is good. I think we can do additional changes after that to make it even better. One of them being highlighting a bit better in the UI which streams are on and not require people to scroll past dozen of deactivated streams.

[evantahler]

Thanks for that, I think the idea is good. I think we can do additional changes after that to make it even better. One of them being highlighting a bit better in the UI which streams are on and not require people to scroll past dozen of deactivated streams.

Exactly! That's why the API is now returning some additional information which can enable that