option to always_delete_completed_jobs_including_failed

[wearpants]

Feature description

in addition to

[load]
delete_completed_jobs=true

it’d be helpful to have an option to always_delete_completed_jobs_including_failed=true

Are you a dlt user?

Yes, I'm already a dlt user.

Use case

in one of my deploys, I’m running dlt every 60s, and repeated failures could pile up and lead to unbounded performance degradation - I’d rather throw the data away & send an alert & let the user manually debug things

Proposed solution

add this option:

[load]
always_delete_completed_jobs_including_failed=true

Related issues

No response

[rudolfix]

@wearpants
if you just want to clean up the working folder fully, just go for

pipeline.drop()

ie. in finally block after load completes. let me know if that is sufficient to solve your problem

[wearpants]

Will pipeline.drop() remove *all* state or just the big debugging blobs? I want to be able to continue to do incremental merges, but I can't have the disk fill up if things fail