RFC: Database sync tool

[macraig]

The purpose of this discussion is to discuss the need for a database syncing tool that syncs data from users' databases to Meilisearch.

[paradoxloop]

This is a huge painpoint for us. Currently we have numerous tables (that have heave write updates) and we a column in each called "last_sent_search" which we poll every hour and push all the ones that have an "updated_at" greater than "last_sent_search":

• This doesnt help us manage deleted which have not worked through yet and just filter on return.
• Places an uncessary burden on db and isn't even real time.

We know we could use postgres triggers (or one of the syncing tools) to achieve sometime more reliable but have not done the engineering work yet. A more native solution for meilesearch would be great.

[macraig]

Feedback from @vaddenz in meilisearch/documentation#2926 (comment)

Usability: Following the project's configuration guide, the project was unable to start and throwed some exceptions. Here is a related issue: long2ice/meilisync#94
Reliability: Seems that meilisync was unable to capture some changes made in the DB
Poor maintenance
As for the sync tool, I'd suggest that MeiliSearch consider officially integrating some common utilized projects, e.g. Canal, Debezium etc

[Ja7ad]

@macraig New sync tool is available with custom features and concurrent sync .

https://github.com/ja7ad/meilibridge

[Ja7ad]

I think instead of writing sync tools we can create a connector destination for existing cdc projects such as debzium, airbyte or estuary.

I'm trying to contribute to this projects to add destination connector Meilisearch.