topic Re: Databricks workflow each task cost in Data Engineering

Databricks workflow each task cost

Prashanth24 — Wed, 31 Jul 2024 15:57:10 GMT

Suppose if we have 4 tasks (3 notebooks and 1 normal python code) in a workflow then i would like to know the cost incurred for each task in the Databricks workflow. Please let me know the any way to find out this details.

Re: Databricks workflow each task cost

Edthehead — Fri, 02 Aug 2024 05:33:27 GMT

If each of the tasks are sharing the same cluster then no, you cannot differentiate the costs between the tasks. However, if you setup each task to have its own job cluster, then pass some custom tags and you can then differentiate/report the costs based on the custom tags. On the Azure portal you will be able to see the costs at a tag level.

Re: Databricks workflow each task cost

Prashanth24 — Mon, 05 Aug 2024 09:03:32 GMT

Thanks for the information. One more question specific to this task cost - there are different types of tasks like Notebook, Python, Wheel, Condition task etc. There will be some code like reading xml, yaml, json etc which requires normal python code and do not want distributed parallel processing like notebook task. So in this case is there any price variation of using different types of task like notebook or python etc.

Re: Databricks workflow each task cost

Edthehead — Mon, 05 Aug 2024 09:38:46 GMT

The charge is by the cluster and does not depend on what is running on it. You may be running a python notebook on 1 driver + 5 nodes, but you will still be charged for the 6 nodes that are running. If you want to reduce the code you will put such tasks to use a separate cluster but on the flip side, it will take time for each individual cluster to spin up. So the best way would probably be to have all your tasks run on the same job cluster config with autoscale on.