Cost of running a job on databricks

[Prem1902]

Hi All, 

I need assistance with the cost of running the job on Databricks where I have 20-30 TB (one-time job) and daily data would be around 2GB. 

The level of transformation would be medium.  Source and destination is AWS s3.

Looking for your quick response .

Thanking in anticipation

[Ayushi_Suthar · Accepted solution]

Hi @Prem1902, Good Day! 

Thank you for raising your concern with us. Can you please check this document on the Billable usage system table. With system tables, you can view your account’s global usage from whichever region your workspace is in.

Document : https://docs.databricks.com/en/administration-guide/system-tables/billing.html#billable-usage-system...

However please note that System Tables gives us the details on a per Job ID basis and not per run ID.

Apart from this another option is to configure overwatch which has jobruncostpotentialfact schema which contains the DBU costs along with the run ID.

jobruncostpotentialfact Data Dictionary : https://databrickslabs.github.io/overwatch/dataengineer/definitions/#jobruncostpotentialfact
Deploy OverWatch : https://databrickslabs.github.io/overwatch/deployoverwatch/

Please let me know if this helps and leave a like if this information is useful, followups are appreciated.
Kudos
Ayushi

[Prem1902]

Is there a way to predict the cost before building the solution? I mean we wanted to see our option on different platforms.