topic Cost of running a job on databricks in Data Engineering

Cost of running a job on databricks

Prem1902 — Mon, 19 Feb 2024 10:11:11 GMT

Hi All,

I need assistance with the cost of running the job on Databricks where I have 20-30 TB (one-time job) and daily data would be around 2GB.

The level of transformation would be medium. Source and destination is AWS s3.

Looking for your quick response .

Thanking in anticipation

Re: Cost of running a job on databricks

Ayushi_Suthar — Mon, 19 Feb 2024 11:23:36 GMT

Hi @Prem1902, Good Day!

Thank you for raising your concern with us. Can you please check this document on the Billable usage system table. With system tables, you can view your account’s global usage from whichever region your workspace is in.

Document : https://docs.databricks.com/en/administration-guide/system-tables/billing.html#billable-usage-system-table-reference

However please note that System Tables gives us the details on a per Job ID basis and not per run ID.

Apart from this another option is to configure overwatch which has jobruncostpotentialfact schema which contains the DBU costs along with the run ID.

jobruncostpotentialfact Data Dictionary : https://databrickslabs.github.io/overwatch/dataengineer/definitions/#jobruncostpotentialfact
Deploy OverWatch : https://databrickslabs.github.io/overwatch/deployoverwatch/

Please let me know if this helps and leave a like if this information is useful, followups are appreciated.
Kudos
Ayushi

Re: Cost of running a job on databricks

Prem1902 — Mon, 19 Feb 2024 19:37:58 GMT

Is there a way to predict the cost before building the solution? I mean we wanted to see our option on different platforms.