You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
There are situations where it's necessary to copy partial data to a table that contains historical series of information, for example, monthly data. In this case, we need to first delete rows from destination table corresponding to that month and, after, use FastETL DbToDbOperator to ingest rows from source.
But this process becomes vulnerable, allowing undesirable effect: inconsistent data.
If copy fails for any reason, data has been already deleted from destination. This may cause dangerous data loss.
In order to avoid it, the function copy_db_to_db should accept pre_copy parameter, passing a SQL statement to execute on destination before copy starts. Tipically, the statement would be a delete command filtering the same monthly data that will be copied after.
Note that SQL command delete (without commit) followed by FastETL copy (without truncate) will form an unique transaction, avoiding inconsistent data.
Besides that, other useful SQL commands may be passed to deal with other cases.
The text was updated successfully, but these errors were encountered:
There are situations where it's necessary to copy partial data to a table that contains historical series of information, for example, monthly data. In this case, we need to first delete rows from destination table corresponding to that month and, after, use FastETL DbToDbOperator to ingest rows from source.
But this process becomes vulnerable, allowing undesirable effect: inconsistent data.
If copy fails for any reason, data has been already deleted from destination. This may cause dangerous data loss.
In order to avoid it, the function
copy_db_to_db
should acceptpre_copy
parameter, passing a SQL statement to execute on destination before copy starts. Tipically, the statement would be adelete
command filtering the same monthly data that will be copied after.Note that SQL command
delete
(without commit) followed by FastETLcopy
(without truncate) will form an unique transaction, avoiding inconsistent data.Besides that, other useful SQL commands may be passed to deal with other cases.
The text was updated successfully, but these errors were encountered: