Topics to tables mapping config removed in 2.0?

[robertotena]

We were trying to upgrade our connectors from version 1.6.x to 2.0, and we noticed the mapping configurations have been removed.

If I am not wrong, the change is here:

So if I understand correctly, in order to map a topic to a BigQuery dataset and table we need to name the topic like dataset:tablename?

So we are coupling the topic name that can have some standards to the naming of the BigQuery dataset and table? That kills any room for flexibility and impedes us upgrading.

Is there a chance we can look into this and add this feature back? What has been the reason to remove it?

[C0urante]

You can use an SMT to route topics to tables; we called this out in the upgrade notes for 2.0.0:

topicsToTables was removed. SMTs should be used route topics to tables.

You might find the RegexRouter SMT useful here; check out the documentation at https://kafka.apache.org/27/documentation.html#connect_included_transformation.

It's highly unlikely we'll be adding that feature back. This type of logic is best accomplished with an SMT.

[C0urante]

For more context on this decision, check out wepay#245.

[robertotena]

Aaah I understand now. I guess I missed it because I did not see that in the README of this project!

Maybe the Regex SMT specifically is not the best for us (as we have many different mappings), but with this approach we can tackle it from the outside.

[C0urante]

I guess I missed it because I did not see that in the README of this project!

You know, that's a fair point 😅 . I've filed #89 to track some updates to the README that we should make, including a note on the removal of the topicsToTables and datasets properties.