refactor: DAG refinement

[joshgamache]

Tasks

Beta Give feedback

We're having trouble rendering your tasklist, it's most likely has one of the following formatting errors:

• indented tasks (i.e. nested tasklist; support for this is coming!)
• an empty task (i.e. - [ ] on a line by itself)
• any empty new lines (before/after the tasklist or in between tasks)
• duplicate issue/pr links
• a draft task exceeds 512 characters

Please check the tasklists documentation for more information. 🔗 Feedback Discussion

Thank you for participating in the Private Beta ❤️

1. A few of the current DAGs have a run-order dependence. It would be good practice to combine these into a single DAG for flow handling
2. DAGs, as they are, currently only load initial data but not subsequent data. A separate _initial load_ DAG should exists, which is then supported by triggered/scheduled DAGs to ingest new data.
3. DAGs are triggered manually. A DAG should be created to monitor the storage dropbox and send to the correct pipeline.