databricks job compute price w.r.t running time

[Raja_fawadAhmed]

I have two workflows (jobs) in data bricks (AWS) with below cluster specs (job base cluster NOT general purpose)

Driver: i3.xlarge · Workers: i3.xlarge · 2-8 workers 

Job 1 takes 10 min to complete

Job 2 takes 50 min to complete

Questions:

DBU cost is same for both the jobs ?

Would the cost reduce if i combine both jobs to one job (merge code and run it as one job) ?

[Kaniz_Fatma]

Hi @Raja_fawadAhmed,

1. DBU Cost for Both Jobs:

   • Databricks pricing is based on DBUs (Databricks Units) consumed. The cost depends on the type of compute instances used and the specific workload.
   • For your two jobs, the DBU cost would be calculated separately based on their resource consumption.
   • To provide an accurate comparison, we need to know the specific DBU rates for each job. These rates can vary based on factors like instance type and workload type.
   • If you have the DBU rates for both jobs, we can calculate the cost individually.

2. Combining Jobs:

   • Combining both jobs into a single job might impact the cost, but it depends on several factors:
     • Overhead: Combining jobs reduces the overhead of starting and stopping clusters, which can be cost-effective.
     • Cluster Configuration: If both jobs have similar resource requirements, merging them could lead to better cluster utilization.
     • Concurrency: If the jobs run concurrently, combining them may not reduce costs significantly.
     • DBU Efficiency: Consider whether the combined job can utilize resources more efficiently than running them separately.
   • Photon Engine: Databricks offers a Photon Engine for massively parallel processing, which can optimize DBU usage. If your workload supports it, using Photon might reduce costs.
   • Pricing Calculator: You can use Databricks’ Pricing Calculator to estimate costs based on your specific scenario.

3. Recommendation:

   • Compare the DBU costs for both jobs individually and then evaluate the impact of combining them.
   • If the combined job can utilize resources efficiently and reduce overhead, merging might be cost-effective.
   • Always consider workload characteristics, concurrency, and resource utilization when making decisions.

Remember that Databricks provides flexibility, so you can adjust your approach based on your specifi...¹².