Some jobs have too many tasks

[astrojuanlu]

https://linen-slack.kedro.org/t/22732083/announcing-kedro-databricks-https-github-com-jenspederm-kedr#0dffcc72-1ac6-4c4f-9bb0-47a654643cd0

I have been trying out your plugin today. It sucks that databricks limits to max of 100 tasks in a single job. I have more than 100 nodes in my default pipeline T.T which is why it is crashing...

A possible solution would be to group tasks, like kedro-airflow does since kedro-org/kedro-plugins#241 cc @ankatiyar, @sbrugman

There are other possible solutions maybe.

I'm wondering, does this "100 tasks per job" limitation depend on the Databricks config/cluster? Or is it universal?

[datajoely]

I'm wondering, does this "100 tasks per job" limitation depend on the Databricks config/cluster? Or is it universal?

My hunch is 1 task is 1 container - the same reason you shouldn't need 28 containers to run spaceflights

[JenspederM]

I'm wondering, does this "100 tasks per job" limitation depend on the Databricks config/cluster? Or is it universal?

My hunch is 1 task is 1 container - the same reason you shouldn't need 28 containers to run spaceflights

Think you're right. As far as I can tell, it's universal. But I haven't really found any mention of it outside of this issue from 2021: https://community.databricks.com/t5/data-engineering/how-many-jobs-can-i-create-in-my-databricks-workspace/td-p/18111