Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add option to copy historical data perserving consistency #185

Open
washolive opened this issue Dec 20, 2023 · 0 comments
Open

Add option to copy historical data perserving consistency #185

washolive opened this issue Dec 20, 2023 · 0 comments

Comments

@washolive
Copy link
Member

There are situations where it's necessary to copy partial data to a table that contains historical series of information, for example, monthly data. In this case, we need to first delete rows from destination table corresponding to that month and, after, use FastETL DbToDbOperator to ingest rows from source.

But this process becomes vulnerable, allowing undesirable effect: inconsistent data.
If copy fails for any reason, data has been already deleted from destination. This may cause dangerous data loss.

In order to avoid it, the function copy_db_to_db should accept pre_copy parameter, passing a SQL statement to execute on destination before copy starts. Tipically, the statement would be a delete command filtering the same monthly data that will be copied after.

Note that SQL command delete (without commit) followed by FastETL copy (without truncate) will form an unique transaction, avoiding inconsistent data.

Besides that, other useful SQL commands may be passed to deal with other cases.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant